NPS-MV- 11-001 


NAVAL 

POSTGRADUATE 

SCHOOL 


MONTEREY,  CALIFORNIA 


ANALYSIS  OF  SYSTEM  TRAINING  IMPACT  FOR 
MAJOR  DEFENSE  ACQUISITION  PROGRAMS 

by 

Robert  A.  Wisher,  Frederiek  E.  Hartman,  Charles  G.  Sanders,  and 

Anthony  Ciavarelli 

August  2011 

Approved  for  public  release;  distribution  is  unlimited 


Prepared  for:  Offiee  of  the  Deputy  Assistant  Seeretary  of 
Defense  (Readiness),  The  Pentagon,  Room  1E537,  Washington,  DC  20301-4000 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


NAVAL  POSTGRADUATE  SCHOOL 
Monterey,  California  93943-5000 


Daniel  T.  Oliver 
President 


Leonard  A.  Ferrari 
Executive  Vice  President  and 
Provost 


This  report  was  prepared  for  and  funded  by  the  Office  of  the  Deputy  Assistant  Secretary  of 
Defense  (Readiness),  The  Pentagon,  Washington,  D.C. 


Further  distribution  of  all  or  part  of  this  report  is  authorized. 


This  report  was  prepared  by: 

Robert  A.  Wisher 
Research  Professor 

Charles  G.  Sanders 
Consultant 


Frederick  E.  Hartman 
Consultant 

Anthony  Ciavarelli 
Consultant 


Reviewed  by: 

CDR  Joseph  Sullivan 
MOVES  Institute 


Released  by: 

Karl  A.  van  Bibber,  Ph.D. 

Vice  President  and  Dean  of  Research 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data 
sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other 
aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information 
Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other 
provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently  valid  OMB  control  number. 
PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 


1.  REPORT  DATE  (DD-mm-yyyy)  2.  REPORT  TYPE 

31-08-2011  Technical  Report 


4.  TITLE  AND  SUBTITLE 


3.  DATES  COVERED  (Fmm-To) 
01-10-2010-07-31-2011 


5a.  CONTRACT  NUMBER 


Analysis  of  System  Training  Impact  for  Major  Defense  Acquisition  Programs 


5b.  GRANT  NUMBER 


6.  AUTHOR(S) 

Robert  A.  Wisher,  Frederick  E.  Hartman,  Charles  G.  Sanders,  and  Anthony 
Ciavarelli 


5c.  PROGRAM  ELEMENT 
NUMBER 


5d.  PROJECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES)  AND  ADDRESS(ES) 
Naval  Postgraduate  School 
MOVES  Institute 
Monterey,  CA  93010 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Office  of  the  Assistant  Secretary  of  Defense,  Readiness 
The  Pentagon,  Room  1E525,  Washington,  DC  20301 


8.  PERFORMING 
ORGANIZATION  REPORT 
NUMBER 


10.  SPONSOR/MONITOR’S 
ACRONYM(S) 


II.  SPONSOR/MONITOR’S 
REPORT  NUMBER(S) 
NPS-MV-1 1-001 


12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 
Approved  for  public  release;  distribution  is  unlimited 


13.  SUPPLEMENTARY  NOTES 

The  views  expressed  in  this  report  are  those  of  the  authors  and  do  not  reflect  the  official  policy  or  position 
of  the  Department  of  Defense  or  the  U.S.  Government. 


14.  ABSTRACT 

The  study  examined  the  time-course  results  of  planning  and  executing  training  from  the  inception  of  a  new  system  to 
its  acquisition  and  support  after  fielding,  in  particular  for  Major  Defense  Acquisition  Programs.  This  study  is  the  first 
phase  in  addressing  these  issues  surrounding  the  impact  of  partial,  inadequate,  misguided,  or  diminished  resources  for 
training  for  new  systems.  This  phase  gathered  evidence  from  the  literature  published  over  the  past  30  years, 
consolidated  that  evidence  into  a  database,  and  searched  for  potential  gaps  between  what  was  done  and  what  could 
have  been  done  given  the  extant  level  of  knowledge  about  training.  Reports  were  identified  and  coded  across  36 
variables.  More  than  4,000  technical  reports  and  500  General  Accountability  Office  reports  were  analyzed  for 
inclusion,  with  more  than  75  being  coded.  A  gap  analysis  was  conducted  which  searched  for  differences  between  an 
^existing  status'  and  a  'potential  status.’  Four  types  of  gaps  were  identified:  knowledge  gaps,  awareness  gaps, 
implementation  gaps,  and  commitment  gaps.  The  analysis  yielded  26  instances  of  gaps,  validating  the  need  for  a  more 
critical  treatment  of  training  in  major  acquisitions.  The  gap  and  trend  analysis  of  this  data  provide  a  start  point  for  the 
second  phase  of  the  study  which,  based  on  the  current  findings,  examines  a  series  of  selected  systems  for  detailed 
training  assessment  through  a  case  study  methodology. 


15.  SUBJECT  TERMS 

training,  readiness,  major  acquisition  programs,  training  evaluation,  training  assessment,  gap  analysis 


16.  SECURITY  CLASSIFICATION  OF: _  17.  LIMITATION 

a.  REPORT  I  b.  ABSTRACT  I  c.  THIS  PAGE  OF  ABSTRACT 


Unclassified  Unclassified 


Unclassified 


18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

Robert  Wisher 
19b.  TELEPHONE 
NUMBER  (include  area  code) 
703-887-0543 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39.18 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


EXECUTIVE  SUMMARY 


Background 

Training  prepares  individuals,  teams,  and  leaders  to  operate  and  maintain  the  vast  array 
of  systems  that  are  being  acquired,  modernized,  updated,  and  enhanced  to  provide 
operational  capabilities.  If  a  human  cannot  be  properly  trained,  the  entire  system  will 
function  sub-optimally.  The  focus  on  the  current  study  is  the  time-course  results  of 
planning  and  executing  training  from  the  inception  of  a  new  system  to  its  acquisition  and 
support  after  fielding,  in  particular  for  Major  Defense  Acquisition  Programs.  This  study 
is  the  first  phase  in  addressing  these  issues  surrounding  the  impact  of  partial,  inadequate, 
misguided,  or  diminished  resources  for  training  for  new  systems. 


Approach 

The  purpose  of  this  phase  is  to  gather  evidence  from  the  literature  published  over  the  past 
30  years,  consolidate  that  evidence  into  a  database,  and  search  for  potential  gaps  between 
what  was  done  and  what  could  have  been  done  given  the  extant  level  of  knowledge  in  the 
training  community.  This  first  phase  begins  to  lay  the  foundation  to  quantify  a 
relationship  between  the  quality  of  training  and  its  effect  on  optimizing  total  systems 
performance.  Reports  were  identified  in  the  published  literature  and  coded  across  36 
variables.  More  than  4,000  technical  reports  and  500  General  Accountability  Office 
reports  were  analyzed  for  inclusion,  with  more  than  75  being  coded. 

Findings 

A  gap  analysis  was  conducted  which  searched  for  differences  between  an  ‘existing 
status'  and  a  ‘potential  status'  or  what  might  be  the  desired  status.  Four  types  of  gaps 
were  identified:  knowledge  gap  -  a  best  training  practice  has  not  been  fully  validated  or 
proven  for  application  for  a  given  training  system.;  awareness  gap  -  a  best  training 
practice  is  proven,  established,  and  relevant,  but  has  not  been  applied  for  the  training 
system  of  interest;  implementation  gap  -  a  valid  best  training  practice  has  been  identified 
and  attempted,  but  did  not  work  properly  in  the  case  of  a  given  training  system, 
commitment  gap  -  a  valid  best  training  practice  is  recognized  but  not  applied  due  to 
policy,  cost,  schedule  or  other  factors.  The  phase  one  analysis  yielded  26  instances  of 
gaps,  validating  the  need  for  a  more  critical  treatment  of  training  in  major  acquisitions. 
The  gap  and  trend  analysis  of  this  data  provide  a  start  point  for  the  second  phase  of  the 
study  which,  based  on  the  current  findings,  examines  a  series  of  selected  systems  for 
detailed  training  assessment  through  a  case  study  methodology. 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


IV 


TABLE  OF  CONTENTS 


I.  INTRODUCTION .  I 

II.  TRAINING  PERSPECTIVE .  2 

A.  TRAINING  SIMULATION .  2 

B.  INSTRUCTIONAL  SYSTEMS  DEVELOPMENT  (ISD) .  4 

C.  HUMAN  SYSTEMS  INTEGRATION  (HSI) .  5 

D.  MANPOWER  AND  PERSONNEL  INTEGRATION .  5 

III.  ACQUISITION  PHASES .  7 

A.  MILESTONE  A .  8 

B.  MILESTONE  B .  8 

C.  MILESTONE  C .  8 

D.  POST  MILESTONE  C .  9 

E.  FRONT  END  ANALYSIS .  9 

IV.  OVERVIEW  OF  PHASE  I  PROJECT . 10 

A.  GOALS . II 

B.  KEY  CONCEPTS . II 

C.  RESEARCH  QUESTIONS . II 

V.  RESEARCH  METHODOLOGY .  II 

A.  STEP  I  -  UNDERSTANDING  THE  TRAINING  ENVIRONMENT..  13 

B.  STEP  2-  HOLISTIC  VIEW  OF  THE  ACQUISITION  AND 

TRAINING  ENVIRONMENT . 13 

C.  STEP  3-  GAP  ANALYSIS  FRAMEWORK . 13 

D.  STEP  4  -  DATA  FOR  GAP  ANALYSIS . 13 

E.  APPROACH . 14 

F.  REFINEMENT  OF  VARIABLES . 14 

G.  SOURCE  DOCUMENTS . 16 

H.  CODING  DATABASE . 17 

VI.  RESULTS .  19 

A.  DESCRIPTIVE  STATISTICS .  19 

B.  ANALYSIS  OF  GAPS .  20 

C.  CROSSWALK  WITH  TECHNICAL  REPORT  DATABASE .  24 

D.  FRONT  END  ANALYSIS  AND  GAPS .  24 

VII.  DISCUSSION  AND  CONCLUSIONS . 25 

A.  CLOSING  COMMENTS .  28 

LIST  OF  REFERENCES .  30 


V 


APPENDIX  A  -  POLICY  AND  GUIDANCE  EXCERPTS .  31 

APPENDIX  B  -  CODING  RULES .  40 

APPENDIX  C  -  SAMPLED  TECHNICAL  ABSTRACTS .  43 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


THIS  PAGE  INTENTIONALLY  LEET  BLANK 


I.  INTRODUCTION 


The  goal  of  training  is  to  provide  military  members  with  the  capacity  to  perform 
military  tasks,  both  individually  and  collectively.  Military  training  is  seen  as  a  disciplined 
approach  to  preparing  individual  warfighters  and  units  for  military  operations.  Training 
is  about  exercising  troops  in  the  field,  training  staffs  at  their  operations  centers,  training 
aircrews  while  in  real  or  simulated  flight,  or  training  maintenance  technicians  on  brand 
new  or  decades  old  systems.  Training,  then,  prepares  individuals  and  teams  to  operate 
and  maintain  the  vast  array  of  systems  that  are  being  acquired,  modernized,  updated,  and 
enhanced  to  provide  operational  capabilities.  If  a  human  cannot  be  properly  trained,  the 
entire  system  will  function  sub-optimally. 

The  focus  on  the  current  study  is  the  time  course  of  planning  and  executing 
training  from  the  inception  of  a  new  system  to  its  acquisition  and  support  after  fielding,  in 
particular  for  Major  Defense  Acquisition  Programs  (MDAPs)  which  are  defined  as  those 
costing  in  excess  of  2  billion  dollars.  How  adequate  are  the  training  plans,  what  is  their 
priority  as  the  system  evolves,  do  they  take  advantage  of  known  advances  in  training  that 
others  have  proved  to  be  effective?  This  study  is  the  first  phase  in  addressing  these  issues 
with  a  longer  view  to  understanding  the  impact  of  partial,  inadequate,  misguided,  or 
diminished  resources  for  and  approaches  to  training  for  new  systems.  The  purpose  of  this 
phase  is  to  gather  evidence  from  the  literature  published  over  the  past  30  years, 
consolidate  that  evidence  into  a  database,  and  search  for  gaps  between  what  was  done 
and  what  could  have  been  done  given  the  level  of  knowledge  in  the  training  community. 
This  first  phase  begins  to  lay  the  foundation  to  quantify  a  relationship  between  the  quality 
of  training  and  its  effect  on  optimizing  total  systems  performance. 

By  itself,  this  study  does  not  answer  fully  the  issue  at  hand;  rather  it  is  the 
beginning  of  a  multi-phased  approach  to  systematically  investigate,  document,  and 
conclude  what  the  relationship  is.  Later  phases  may  revise  or  expand  the  database, 
conduct  case  studies,  and  explore  quantitative  techniques  that  can  clearly  elucidate  the 
many  aspects  of  the  problem  and  how  they  tie  together.  This  study,  then,  is  the  beginning 
of  that  endeavor  and  it  starts  with  collecting  and  organizing  relevant  reports  from  sources 
internal  and  external  to  DoD. 

The  report  first  develops  a  training  perspective  and  discusses  the  impact  of 
simulation  and  instrumentation  on  training  practices  in  recent  decades.  Continuing  the 
background  discussion,  the  role  of  instructional  systems  design  is  then  covered,  followed 
by  the  advocacy  of  training  and  other  people -related  consideration  in  system  acquisition. 
The  acquisition  process  is  then  summarized  and  policies  that  impact  training  are  also 
summarized.  The  study  itself  is  then  presented  with  methods,  data  collection  and 
analysis,  and  the  essential  findings.  We  then  offer  how  this  study  forms  the  foundation 
for  future  work  and  what  that  work  can  be.  Appendices  supplement  the  main  body  of  the 
report. 
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II.  TRAINING  PERSPECTIVE 


Gorman  (1990)  in  his  report,  The  Military  Value  of  Training,  discusses  the  value 
of  training  in  terms  of  historical  “combat  exposure”  data  drawn  from  air  combat  fighter 
engagements  from  World  War  I  and  II,  the  Korean  conflict,  and  Viet  Nam.  In  all  these 
wars,  the  data  show  that  a  relatively  few  number  of  participants  account  for  most  of  the 
successful  engagement  outcomes  -  or  enemy  kills.  The  data  from  wartime  statistics 
showed  clearly  that  those  warriors  who  survived  the  first  5  or  so  combat  engagements 
went  on  to  engage  and  succeed  in  producing  most  of  the  combat  kills.  Such  data  led  to 
strategies  that  created  training  systems  designed  to  replicate  wartime  engagement 
experiences  -  or  in  the  words  of  the  warrior-  you  train  like  you  will  fight. 

Examples  often  used  to  describe  highly  realistic  combat  training  include  the 
Fighter  Weapons  School,  known  as  TOPGUN  (Baranek,  2010).  This  school  provides  air- 
to-air  combat  training  using  instrumented  ranges  that  allow  pilots  under  instruction  to  fly 
their  combat  aircraft  in  realistic  engagements  against  professional  aviation  instructors 
who  serve  as  competent  adversaries.  Air  to  air  combat  engagements  are  recorded  and 
played  back  in  order  to  provide  after  action  review  and  expert  tactical  instructor 
performance  feedback  to  pilots  under  training.  As  a  result  of  such  planned,  organized  and 
systematic  application  of  technology  and  explicit  instructional  principles  exemplified  by 
TOPGUN,  the  kill  to  loss  ratio  in  Vietnam  moved  from  3:1  to  12:1  over  a  four  year 
training  period  between  1969  -  1973  (Gorman,  1990). 

Unless  manpower  and  personnel  selection  approaches  can  acquire  and  assign 
highly  qualified,  pre-trained  individuals  directly  from  the  civilian  economy  to  operate  and 
maintain  new  equipment,  the  development  of  training  is  an  imperative.  This  necessity,  of 
course,  has  been  recognized  and  training  has  always  been  a  component  of  system 
acquisition  plans.  Indeed,  training  systems  acquisition  is  an  increasingly  important 
aspect  of  program  acquisition  extending  over  the  full  acquisition  life  cycle. 

A.  TRAINING  SIMULATION 

The  use  of  instrumented  training  ranges,  like  the  US  Navy’s  Tactical  Air  Combat 
Training  System  used  by  TOPGUN  and  the  USAF  Red  Flag  Training  Range  was  closely 
followed  by  the  rise  of  advanced  simulation  training.  The  use  of  advanced  flight 
simulators  also  emphasized  the  principle  of  providing  “realistic  combat  training.”  A  key 
aspect  of  simulation,  over  instrumented  ranges  was  the  expectation  of  reducing  training 
cost  by  using  simulators  instead  of  actual  operational  aircraft  or  other  expensive  weapon 
platforms.  But  the  use  of  simulation  during  the  70’s  and  80’s  in  particular  raised  certain 
questions  and  concerns  about  technical  issues  surrounding  the  degree  of  realism  that  such 
simulation  could  deliver.  Some  of  the  main  issues  and  concerns  that  arose  during 
development  and  application  of  simulation  training  were  summarized  and  discussed  in  a 
report  published  by  the  Institute  for  Defense  Analysis  entitled  The  Value  of  Simulation 
for  Training.  Some  of  the  key  issues  are  as  briefly  summarized  below: 
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•  Can  the  simulation  constructed  actually  represent  the  actual  equipment  and/or 
operating  environment  with  sufficient  “fidelity”  -  or  in  the  eyes  of  the  operator 
and  maintainer  -  is  the  simulation  “realistic”  enough  for  the  training  required? 

•  How  do  the  relative  costs  and  effectiveness  of  simulators  compare  to  simply  using 
the  actual  equipment  in  its  real  -  world  environment? 

•  Can  a  given  simulation  device  be  used  to  reduce  the  time  or  number  of  training 
events  required  to  achieve  the  same  training  objectives  in  the  actual  equipment? 

•  What  are  some  of  the  best  practices  in  using  simulation?  Such  as  (1)  the  proper 
balance  between  the  use  of  simulation  and  live  training  ranges,  (2)  the  appropriate 
allocation  of  training  time  between  part  task  and  full  mission  simulation  devices. 

•  Can  certain  exclusions  and  limitations  be  employed  in  simulation  with  minimum 
impact  on  training  effectiveness  (for  example,  lack  of  motion  in  flight  simulation 
or  limited  visual  field  of  view,  and  or  display  resolution  for  visual  flight 
simulation? 

•  What  is  the  most  appropriate  type  and  level  of  training  management  and 
instructional  support  required  to  facilitate  device  utilization  and  instructional 
quality  (Orlansky,  Dahlman,  Hammon,  Metzko,  Taylor,  and  Youngblut,  1994). 

The  authors  (Orlansky,  et  al  1994)  conclude  that  flight  and  maintenance 
simulators  train  as  well  as  the  actual  equipment  and  typically  cost  less  to  produce  and 
operate  than  the  actual  equipment.  Another  important  finding  was  that  across  simulation 
domains,  most  problems  were  not  attributed  to  the  lack  of  operational  realism  or 
simulation  fidelity  but  rather  were  due  to  poor  training  development  planning,  the 
absence  of  instructional  support  and  training  features  on  the  simulation  device  (such  as 
recording  and  playback). 

Finally,  it  is  clear  from  the  information  provided  in  the  Orlansky  study’s  literature 
review  and  other  studies  that  the  effectiveness  of  a  training  device  depends  less  on  the 
simulation  technology  used  and  more  on  the  quality  of  the  training  plan  and  how  the 
simulation  training  process  is  actually  supported  and  managed.  See  also  Stewart,  J.E., 
Johnson,  D.M,  &  Howse,  W.R.  (2008)  for  more  detail  on  this  subject.  There  are  some 
persistent  issues  associated  with  the  development  and  application  of  such  technology  - 
based  training  systems  that  may  impact  training  effectiveness,  such  as: 

1 .  Training  need  is  not  recognized  or  requirement  is  misunderstood. 

2.  Training  need  is  recognized  but  is  not  supported  because  of  policy  or  resource 
limitations. 

3.  Technology  needed  for  training  is  not  mature  enough  or  otherwise 
unavailable. 

4.  Training  System  is  implemented  but  it  is  not  properly  managed  or  used 
effectively. 

5.  Training  System  development  lags  behind  the  weapon  system  operational 
implementation. 

6.  The  Training  System  implemented  is  poorly  designed  /or  otherwise  cannot 
meet  the  operational  training  requirements. 
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B.  INSTRUCTIONAL  SYSTEMS  DEVELOPMENT  (ISD) 


With  the  rise  of  advaneed  training  simulation  application  in  the  70’s  and  80’s, 
there  was  a  concerted  attempt  to  broadly  apply  various  “systems  approaches”  to  the 
design,  development  and  evaluation  of  military  training  systems.  During  this  period  the 
US  military  training  establishment  ultimately  created  a  uniform  standard  referred  to  as 
the  Interservice  Instructional  Systems  Development  (ISD)  procedure  (Branson,  1978). 

In  brief,  ISD  was  supposed  to  provide  a  means  to  drive  training  design  and 
development  through  the  use  of  a  phased  systematic  process  that  began  with  detailed 
requirements  analysis  (including  trainee  knowledge  and  skill  specification)  and  finished 
with  specific  training  evaluations  dedicated  to  validating  the  effectiveness  of  the  training 
system.  However,  after  ten  years  of  application  of  the  ISD  model  in  military  training,  it 
became  apparent  that  simply  having  a  systematic  framework  with  its  detailed  procedural 
process  was  not  enough  to  ensure  effective  training.  McCombs,  B.L.  (1986)  reviewed 
literature  on  the  origin,  application  and  effectiveness  of  ISD  Analysis,  Design, 
Development,  Implementation,  and  Evaluation  -  or  the  so-called  ADDIE  model. 
McCombs  concluded  that  the  use  of  ISD  per  se  did  not  necessarily  result  in  the  most 
effective  training. 

Eor  example  a  serious  problem  with  ISD  was  the  assumption  that  people  using  the 
approach  have  the  requisite  knowledge  and  skills  to  appropriately  and  effectively  apply 
the  process.  Unfortunately,  this  has  not  been  the  case.  McCombs  argued  that  often  the 
people  implementing  ISD  did  not  fully  understand  the  training  requirement,  or  lacked  the 
technical  skills  necessary  to  apply  the  methods  specified  in  the  ISD  process  (such  as  task 
analysis,  skill  specification,  and  evaluation  methods). 

Another  ISD  issue  is  that  the  process  itself  does  not  provide  sufficient  detail  to 
map  out  valid  instructional  strategies  and  media  selection  prescriptions,  based  on  a 
scientific  understanding  of  specific  learning  domains  or  complex  tasks.  Eor  an  updated 
description  of  the  ADDIE  model  and  its  application  in  education  and  training  see  Hodell 
(2000)  ISD  from  the  Ground  Up.  And  for  an  updated  military  equivalent  description  see 
also  Department  of  Defense  (2001)  Instructional  Systems  Development/Systems 
Approach  to  training  and  education  (MIE-HDBK-29612-2A),  which  applies  to  training 
development  for  new  systems. 

Training  is  often  joined  with  other  ‘people  factors’  and  variably  termed 
human  systems  integration  (HSI)  or  by  Service  programs,  such  as  the  overarching 
Army  effort  known  as  MANPRINT,  or  manpower,  personnel  and  training 
integration  program.  Current  acquisition  policy  requires  that  a  program  manager 
have  a  plan  for  HSI  in  place  early  in  the  acquisition  process  to  “ensure  that  the 
system  is  built  to  accommodate  the  characteristics  of  the  user  population  that  will 
operate,  maintain,  and  support  the  system.”  Further  requirements  are  that 
“human-factors  engineering  to  design  systems  that  require  minimal  manpower; 
provide  effective  training;  can  be  operated  and  maintained  by  users;  and  are 
suitable  (habitable  and  safe  with  minimal  environmental  and  occupational  health 
hazards)  and  survivable  (for  both  the  crew  and  equipment)”  (DoDI  5002.2,  2008). 
Program  managers  are  required  to  summarize  the  HSI  plan  in  the  Acquisition 
Strategy  and  the  Systems  Engineering  Plan.  Extracts  from  relevant  DoD  policies 
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as  issued  in  various  directives,  instructions,  and  regulations  are  presented  in 
Appendix  A. 


C.  HUMAN  SYSTEMS  INTEGRATION  (HSI) 

The  concept  of  Human  Systems  Integration  emerged  in  the  1990’s  and  is  the 
current  prevailing  systems  engineering  framework  now  promulgated  by  the  Department 
of  Defense.  The  founding  premise  of  HSI  as  stated  by  (DoD  5000.02:  Operation  of  the 
Defense  Acquisition  System,  2009)  is  as  follows: 

The  Program  Management  (PM  )  shall  have  a  plan  for  HSI  in  place  early  in  the 
acquisition  process  to  optimize  total  system  performance,  minimize  total  ownership  costs, 
and  ensure  that  the  system  is  built  to  accommodate  the  characteristics  of  the  user 
population  that  will  operate,  maintain,  and  support  the  system. 

Human  Systems  Integration  (HSI)  refers  to  the  planning  and  integration  of  such 
areas  as  manpower,  personnel,  training,  human  factors  engineering,  system  safety, 
personnel  survivability,  and  health  hazards  -  thereby  clearly  reflecting  the  origins  of  the 
systems  framework  originally  outlined  by  MANPRINT  (Booher,  1990,  Clark  and 
Goulder,  2002). 

An  important  goal  of  HSI  is  to  influence  acquisition  decisions  early  in  the 
Defense  acquisition  management  framework.  In  particular,  HSI  solutions  need  to  be 
identified  early  enough  to  allow  changes  in  design.  For  example,  40  percent  of  the  Life 
Cycle  Cost  (LCC)  is  determined  by  Milestone  A.  LCC  is  the  total  cost  to  the  government 
of  acquisition  and  ownership  of  that  system  over  its  useful  life(Office  of  Naval  Research, 
2011). 

With  the  brief  overview  here,  we  have  come  full  circle  with  attempts  to  integrate 
training  into  the  system  acquisition  process,  beginning  with  ISD,  then  MANPRINT  and 
now  HSI  all  emphasizing  early  and  systematic  consideration  to  personnel  training  within 
a  “systems  engineering”  framework  that  addresses  all  of  the  design  and  logistical  support 
functions  thought  to  be  essential  in  creating  and  effective  weapon  system. 

The  program  manger  is  required  to  work  with  the  training  community  to  develop 
options  for  individual  and  collective  training  for  operators,  maintainers,  and  support 
personnel.  Training  decisions  need  to  be  based  on  training  effectiveness  evaluations  to 
maintain  skill  proficiencies  and  reduce  individual  and  collective  training  costs.  The  PM 
shall  develop  training  system  plans  to  maximize  the  use  of  new  learning  techniques, 
simulation  technology,  embedded  training  and  distributed  learning  (DoD  Instruction 
1322.26),  and  instrumentation  systems  that  provide  “anytime,  anyplace”  training  and 
reduce  the  demand  on  the  training  establishment.  Where  possible,  the  PM  shall  maximize 
the  use  of  simulation-supported  embedded  training,  and  the  training  systems  shall  fully 
support  and  mirror  the  interoperability  of  the  operational  system  (DoD  Directive 
1322.18). 

D.  MANPOWER  AND  PERSONNEL  INTEGRATION  (MANPRINT) 

The  Manpower  and  Personnel  Integration  (MANPRINT)  program  initiated  in  the 
1980’s  is  a  more  comprehensive,  attempt  to  apply  a  “systems  approach”  to  weapon 
system  acquisition  by  providing  a  framework  that  calls  for  early  consideration  of  key 
system  planning  activities  and  development  requirements.  The  founding  premise  of 
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MANPRINT  holds  that  the...  The  system  must  be  designed  so  that  the  specified  target 
population  can  be  easily  trained  to  perform  to  standard  (p.6) 

MANPRINT  is  a  comprehensive  management  and  technical  effort  designed  to 
assure  integration  of  all  relevant  information  from  MANPRINT  domains.  The  primary 
goals  of  MANPRINT  are  to  develop  the  most  effective  system  and  to  achieve 
effectiveness  at  lower  life  cycle  costs.  (Metzler,  and  Lewis  1989,  p.I)  MANPRINT 
considers  seven  key  areas  in  the  process  of  system  acquisition  (US  Army  2001)  -  with 
Training  as  the  third  area: 

Seven  MANPRINT  areas: 

1)  Manpower  -  The  number  of  personnel,  both  military  and  civilian  required, 
authorized  and  potentially  available  to  operate,  maintain,  and  support  each  system 
acquisition. 

2)  Personnel  -  The  human  aptitudes,  skills,  and  capabilities  required  to  operate, 
maintain,  and  support  a  system  in  peacetime  and  war. 

3)  Training  -  The  instruction  and  resources  required  to  provide  [Army]  personnel 
with  requisite  knowledge,  skills,  and  abilities  to  properly  operate,  maintain,  and 
support  [Army]  systems. 

4)  Human  Factors  Engineering  -  The  comprehensive  integration  of  human 
capabilities  and  limitations  into  system  definition,  design,  and  evaluation  to 
promote  effective  soldier  -  machine  integration  for  optimal  total  system 
performance. 

5)  System  Safety  -  The  design  and  operational  characteristics  of  a  system  that 
minimize  the  possibilities  for  accidents  or  mishaps  caused  by  human  error  or 
system  failure. 

6)  Health  Hazards  -  The  systematic  application  of  biomedical  knowledge,  early  in 
the  acquisition  process,  to  identify,  assess  and  minimize  health  hazards  associated 
with  the  system’s  operation,  maintenance,  repair  or  storage. 

7)  Soldier  Survivability  -  The  characteristics  of  a  system  that  reduce  fratricide  as 
well  as  reduce  detectability  of  the  soldier,  prevent  attack  if  detected,  prevent 
damage  if  attacked,  minimize  medical  injury  if  wounded  or  otherwise  injured,  and 
minimize  physical  and  mental  fatigue  (p.8) 

MANPRINT,  in  part,  was  developed  as  a  consequence  of  documented  problems 
found  in  weapon  system  acquisition  due  to  a  failure  to  properly  integrate  Manpower, 
Personnel,  and  Training  (MPT)  into  the  weapon  system.  For  example  study  completed 
by  the  General  Accounting  Office  (GAO)  in  1985,  p.  2,  raised  the  following  issues: 

•  No  one  was  in  charge  of  implementing  MPT  integration 

•  MPT  requirements  were  not  being  identified  early  enough  in  the  acquisition 
process  to  influence  system  design 

•  The  Army  was  not  clearly  defining  weapon  system  MPT  needs  for  the  developers 

In  this  1985  study,  the  GAO  concluded  that  The  Army  must  effectively  integrate 
manpower,  personnel  and  training  into  its  weapon  systems  acquisition  process  to  ensure 
that  when  the  weapon  systems  are  deployed,  they  can  be  satisfactorily  operated  and 
maintained  (p.  1 ) 
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It  is  one  purpose  of  this  current  study  to  determine  how  many  of  these  problems, 
and  other  possible  issues  remain  after  26  years  -  given  the  expected  improvements  in  the 
acquisition  process  following  the  introduction  and  implementation  of  MANPRINT. 


III.  ACQUISITION  PHASES 

The  acquisition  of  new  systems,  of  course,  progress  through  phases.  The  DoD 
acquisition  model  encompasses  five  phases  related  to  pre-systems  acquisition  activity,  the 
formal  system  acquisition,  and  the  necessary  sustainment  phase.  Training  is  conceived, 
planned,  developed,  and  implemented  throughout  these  phases.  Three  Milestone 
Decision  Reviews  (A,  B,  and  C)  accompany  the  phases.  The  five  phases  are: 

•  Concept  Refinement  -During  this  phase  a  study  called  an  analysis  of  alternatives 
is  conducted  to  assess  alternatives  to  provide  the  desired  capability  identified  in 
the  Initial  Capabilities  Document  (ICD).  To  achieve  the  best  possible  system 
solution,  materiel  solution  analysis  places  emphasis  on  innovation  and 
competition  and  on  existing  commercial  off-the-shelf  and  other  solutions  drawn 
from  a  diverse  range  of  large  and  small  businesses.  An  analysis  of  alternatives  and 
a  technology  development  strategy  are  developed  to  help  guide  the  efforts  during 
the  next  phase,  which  is  technology  development.  Materiel  solution  analysis  ends 
when  materiel  solution  to  the  capability  need  identified  in  the  ICD  is 
recommended  by  the  lead  component. 

•  Technology  Development  -  This  phase  addresses  the  set  of  technologies  to  be 
integrated  into  a  full  system,  including  the  training  capability.  This  phase 
depends  on  the  system  developer  having  productive  relationships  with  both  the 
user  and  the  science  and  technology  community.  The  process  is  continuous  and 
iterative,  refining  user  requirements  while  assessing  the  maturity  and  feasibility  of 
technologies. 

•  System  Development  and  Demonstration  -  The  acquisition  is  formally  initiated 
and  a  Program  Manager  assumes  responsibility  for  system  development. 
Transition  to  this  phase  requires  approval  of  a  capability  development  document, 
which  reflects  a  technically  mature  capability  that  is  logistically  supportable.  This 
phase  develops  a  system  or  an  increment  of  capability,  completing  full  system 
integration.  Operational  supportability  is  ensured  with  particular  attention  to 
minimizing  the  logistics  footprint  and  implement  human  systems  integration  to 
include  training. 
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•  Production  and  Development  -  The  purpose  of  this  phase  is  to  achieve  an 
operational  capability  that  satisfies  mission  needs.  Operational  test  and  evaluation 
shall  determine  the  effectiveness  and  suitability  of  the  system.  Milestone  C 
authorizes  entry  into  low  rate  initial  production  (for  MDAPs  and  major  systems), 
into  production  or  procurement  for  non-major  systems  that  do  not  require  low  rate 
initial  production,  or  into  limited  deployment  in  support  of  operational  testing. 

•  Operations  and  Support  -During  this  phase,  full  operational  capability  is 
achieved,  each  element  of  logistics  support  is  evaluated  (e.g.,  supply, 
maintenance,  training,  technical  data,  support  equipment),  and  operational 
readiness  is  assessed.  Logistics  and  readiness  concerns  dominate  this  phase.  The 
operations  and  support  phase  includes  life  cycle  sustainment  and  disposal. 

Milestone  A  occurs  upon  the  approval  of  the  Concept-Refinement  Stage  and 
Milestone  B  upon  approval  to  proceed  into  the  Technology  Development  Phase.  The 
achievement  of  Milestone  C  launches  the  production  and  deployment  phase. 

A.  MILESTONE  A:  ENTRY  INTO  TECHNOLOGY  DEVELOPMENT 
PHASE 

The  purpose  of  this  phase  is  to  reduce  technology  risk  and  to  determine  the 
appropriate  set  of  technologies  to  be  integrated  into  a  full  system.  The  project  shall  enter 
Technology  Development  at  Milestone  A  when  the  MDA  has  approved  the  Technology 
Development  Strategy. 

B.  MILESTONE  B:  ENTRY  INTO  SYSTEM  DEVELOPMENT  AND 
DEMONSTRATION  (SDD)  PHASE 

The  purpose  of  the  SDD  phase  is  to  develop  a  system  or  an  increment  of 
capability;  reduce  integration  and  manufacturing  risk  (technology  risk  reduction  occurs 
during  Technology  Development);  ensure  operational  supportability  with  particular 
attention  to  reducing  the  logistics  footprint;  implement  human  systems  integration  (HSI); 
design  for  producibility;  ensure  affordability  and  the  protection  of  critical  program 
information  (CPI);  and  demonstrate  system  integration,  interoperability,  safety,  and 
utility. 

This  effort  is  intended  to  demonstrate  the  ability  of  the  system  to  operate  in  a 
useful  way  consistent  with  the  approved  Key  Performance  Parameters. 

The  completion  of  this  phase  is  dependent  on  a  decision  by  the  MDA  to  commit 
to  the  program  at  Milestone  C  or  a  decision  to  end  this  effort. 

C.  MILESTONE  C:  ENTRY  INTO  PRODUCTION  AND 
DEPLOYMENT  PHASE 

The  purpose  of  the  Production  and  Deployment  phase  is  to  achieve  an  operational 
capability  that  satisfies  mission  needs.  Operational  test  and  evaluation  shall  determine 
the  effectiveness  and  suitability  of  the  system.  Milestone  C  authorizes  entry  into  Low 
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Rate  Initial  Production  (for  MDAPs  and  major  systems),  into  production  or  procurement 
(for  non-major  systems  that  do  not  require  LRIP)  or  into  limited  deployment  in  support  of 
operational  testing  for  MAIS  programs  or  software-intensive  systems  with  no  production 
components.  This  effort  delivers  the  fully  funded  quantity  of  systems  and  supporting 
materiel  and  services  for  the  program  or  increment  to  the  users.  During  this  effort,  units 
shall  attain  Initial  Operational  Capability. 

D.  POST  MILESTONE  C:  OPERATIONS  AND  SUPPORT 

The  objective  of  this  activity  is  the  execution  of  a  support  program  that  meets 
operational  support  performance  requirements  and  sustains  the  system  in  the  most  cost- 
effective  manner  over  its  total  life  cycle.  When  the  system  has  reached  the  end  of  its 
useful  life,  it  shall  be  disposed  of  in  an  appropriate  manner.  Operations  and  Support  has 
two  major  efforts:  Sustainment  and  Disposal. 

Over  the  last  several  decades  the  DoD  has  initiated  several  efforts  aimed  at 
process  update,  and  the  increased  efficiency  to  be  found  in  a  streamlined  process  for 
major  systems  acquisition.  Consistently  during  this  time  the  user/operators  have 
observed  that  the  acquisition  process  is  too  slow,  too  cumbersome  in  terms  of  the 
quantity  and  volume  of  documents  and  in  the  end  provides  some  systems  that  by  the  time 
they  are  fielded  are  out  of  date  with  technology  once  they  emerge  from  the  long  arduous 
process.  It  should  be  recognized  that  technology  insertion  is  very  dependent  on  the  class 
of  acquisition  with  systems  with  intense  software  and  communications  dependencies 
being  the  most  likely  to  suffer  from  the  decades  long  process. 

E.  FRONT  END  ANALYSIS 

The  Defense  acquisition  process  consists  of  the  phases  and  milestones  described 
in  earlier  paragraphs.  During  the  early  or  “front  end”  of  the  acquisition  process,  the 
Concept  Refinement  Phase,  incorporates  detailed  analysis  including  an  analysis  of 
alternatives  to  assess  various  materiel,  software,  and  technologies  solutions  to  shape 
capabilities  to  best  meet  the  military  operational  requirements.  A  recent  topic  of 
discussion  in  Defense  Leadership  meetings  (Hartman,  2009)  has  been  the  need  for  more 
rigorous  assessments  in  the  earliest  stages  of  the  acquisition  process.  Assessment  in  the 
earliest  stages  can  be  most  productive  in  effectively  and  efficiently  meeting  the 
warfighter  needs.  It  is  further  believed  that  early  incorporation  of  system  training  details 
in  the  pre-milestone  A  analysis  has  great  benefit  in  achieving  the  best  possible  system 
solution  and  can  positively  impact  each  stage  of  the  acquisition  life  cycle.  Particularly 
those  systems  with  significant  human  interface  requirements  are  found  to  benefit  from 
bringing  in  training  systems  planning  in  the  concept  refinement  process  in  a  series  of 
iterations  with  the  end  users.  Early  systems  prototypes  (constructive  or  virtual 
simulations)  for  instance  can  be  a  very  effective  tool  in  narrowing  down  and  shaping  the 
final  desired  capability  with  the  warfighters.  The  use  of  modeling  and  simulation  and 
incorporation  of  a  systems  engineering  approach  to  systems  training  can  logically  drive 
the  future  system  training  tools,  simulators  and  part-task  trainers,  as  appropriate.  In 
major  acquisition  programs,  the  ability  to  insert  training  early  in  the  program  can  help 
shape  the  final  quantities  and  technical  complexity  of  the  end  capability.  From  a  senior 
DoD  management  perspective,  proper  front  end  analysis  to  include  the  systems  training 
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considerations  can  permit  a  go  or  no-go  decision  and/or  justify  significant  modifications 
prior  to  the  commitment  of  large,  long  term  fiscal  resources. 


IV.  OVERVIEW  OF  PHASE  I  PROJECT 

This  research  addresses  key  topics  raised  in  the  recent  Strategic  Plan  for  the  Next 
Generation  Training  for  the  Defense  Department,  issued  by  the  Office  of  the  Deputy 
Assistant  Secretary  of  Defense  (Personnel  and  Readiness)  and  signed  by  the  Deputy 
Secretary  of  Defense  on  September  23,  2010.  In  particular,  the  call  for  action  to  develop 
training  capabilities  and  regimens  before  a  system  is  fielded  to  improve  initial  operational 
effectiveness  of  new  capabilities  (§5.6.2)  and  the  call  to  treat  the  training  capabilities  of 
new  systems  with  the  same  rigor  as  their  operational  capabilities  and  updating  as 
appropriate  (§5.6.3)  are  relevant  to  the  current  study.  Training  is  required  to  bridge  the 
gap  between  the  operators,  maintainers,  and  leaders  existing  level  of  knowledge  and  that 
level  required  to  operate  and  maintain  the  system. 

An  underlying  theme  of  the  study  is  to  estimate  the  role  of  training  on  optimizing 
total  systems  performance  for  major  acquisitions.  Traditionally,  training  for  major  new 
systems  is  not  fully  provided  until  after  initial  operational  capability.  However,  if  the 
human  operator  of  the  system  cannot  perform  efficiently,  the  entire  system  will  not 
function  optimally.  Training  is  but  one  factor  here.  The  purpose  of  this  study  is  to 
assess  the  value  of  early  consideration  of  Acquisition  System  training  requirements  for 
new  systems  being  acquired,  or  being  considered,  by  the  Department  of  Defense.  The 
study  is  to  be  based  on  evidence  gathered  in  the  research  and  analysis  literature,  dating  to 
systems  acquired  from  1976  to  the  present  day. 

The  two  central  research  questions  are:  1)  Is  there  a  benefit  to  the  warfighters 
(operators,  maintainers  and  leaders)  and  the  acquisition  community  with  early 
consideration  of  systems  training  in  major  acquisitions?  and  2)  Does  early  system 
training  contribute  to  initial  readiness  and  full  utilization  of  system’s  capability  upon 
initial  delivery?  In  constructing  a  database  to  address  these  specific  questions,  other 
relationships  of  interest  can  emerge.  For  example,  research  that  documents  the  success  of 
a  particular  training  approach  relevant  to  a  specific  system  may  have  been  available,  but 
was  neglected  in  planning  the  training  system  requirements  of  another.  The  identification 
of  such  gaps  is  pivotal  to  the  Phase  I  study.  The  gaps,  for  example,  can  separate 
occurrences  of  diminished  system  performance  on  the  basis  of  the  training  practices  used 
or  not  used,  serving  as  a  basis  for  improvement. 

This  study  draws  from  historic  documents  and  the  traditional  knowledge  that  the 
systems  acquisition  process,  although  inclusive  of  training  plans,  traditionally  addresses 
training  late  in  the  process  and  doesn’t  incorporate  the  advantages  found  in  going  beyond 
the  instructional  systems  development  process  to  determine  training  system 
characteristics.  The  technology  advances  that  enable  the  ability  to  deliver  learning 
content  in  a  many  forms  of  simulators,  simulations,  games  and  on-line  courses  has 
radically  impacted  the  traditional  ways  of  considering  systems  training  in  the  acquisition 
process.  Longitudinal  analysis  of  key  program  documents  provides  specific  examples. 
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both  positive  and  negative,  regarding  the  need  for  including  training  systems  analysis 
early  in  the  acquisition  and  continuing  throughout  the  system  life  cycle. 

A.  GOALS 

The  goal  of  this  phase  is  to  establish  an  overarching  framework  and  research 
methodology  for  the  study.  The  more  detailed  objectives  of  the  initial  phase  are  to 
identify  source  material,  specify  a  working  set  of  variables,  establish  coding  rules,  and  set 
plans  for  the  data  capture  during  the  second  phase.  The  study  formally  started  on  1 
October  2010  with  the  Naval  Postgraduate  School  as  the  lead  activity,  supplemented  by 
technical  staff  from  the  Institute  for  Defense  Analyses  and  Alion  Science  and 
Technology.  The  study  team  consisted  of  experts  with  backgrounds  in  research 
psychology,  operations  research,  training  analysis,  human  factors  engineering,  and 
system  safety,  totaling  more  than  100  years  of  expertise  within  DoD. 

B.  KEY  CONCEPTS 

At  a  meeting  hosted  by  the  Naval  Postgraduate  School  on  21  October  2010,  the 
study  team  openly  considered  a  wide  range  of  concepts  relevant  to  the  study,  such  as 
system  training  plans,  training  support  plans,  human  systems  integration,  domain 
integration,  cost  training  effectiveness  analysis,  isometric  performance  curves,  DoD  5000 
series  of  instructions,  service  regulations,  training  thresholds,  manpower  factors, 
instructional  methods,  and  many  others.  The  discussion  stimulated  numerous  ideas  for 
refinement  in  planning  the  scope  and  research  methodology  for  Phase  I. 

C.  RESEARCH  QUESTIONS 

1 .  What  is  the  value  of  early  training  system  design,  development, 
implementation  and  a  well-planned  training  program?  (i.e.  “doing  it  right”). 

2.  Are  there  specific  gaps  in  applying  “best  training  practices”  during  the  major 
weapon  system  acquisition  process  that  impact  training  effectiveness,  and/  limit 
operational  capabilities  and  readiness? 

IV.  RESEARCH  METHODOLOGY 

The  study  team  considered  alternative  methods  for  assessing  the  evidence  from 
the  literature.  Developing  a  predictive  model  was  deemed  unsuitable  as  there  was  no 
clearly  defined  dependent  variable  to  predict  in  a  consistent  way  across  a  wide  variety  of 
systems.  The  nature  of  the  evidence  can  sometimes  be  qualitative  and  subjective,  while 
at  other  times  it  can  be  quantitative  and  objective,  such  as  ‘striking  a  target’  or  achieving 
a  certain  level  of  reliability.  Rather  than  pursue  a  purely  quantitative  but  highly  limited 
analysis,  the  study  team  adopted  a  holistic  point  of  view  on  capturing  data  values  from 
the  body  of  published  evidence.  Thus,  at  times  a  documented  deficiency  might  be  poor 
performance  in  operational  testing.  At  other  times,  a  low  level  of  acceptance  by  field 
units  due  to  training  issues  or  a  high  rate  of  safety  mishaps  after  fielding  due  to  human 
error,  human  systems  integration,  or  insufficient  training  would  be  the  documented 
deficiency.  The  key  is  to  link  any  deficiencies  to  a  lag  in  delivering  a  robust  training 
capability  at  the  time  of  initial  delivery  of  the  system.  Such  a  search  for  evidence  from 
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reports  and  other  case  studies  would  be  conducted  on  a  case-by-case  basis  across  the 
spectrum  of  major  systems  acquired  between  1976  and  2010.  The  priority  and  status  of 
training  early  in  the  acquisition  process  is  of  greatest  concern. 

The  study  team  decided  to  focus  first  on  a  comprehensive  collection  of  reports 
and  documents  relevant  to  the  research  questions,  beginning  with  a  sample  of  reports.  A 
preliminary  set  of  variables  were  nominated  within  the  first  two  weeks  of  the  study. 
Furthermore,  the  team  saw  value  in  identifying  gaps  in  what  the  system  training  offered 
compared  to  what  it  could  have  offered  as  documented  elsewhere  in  the  literature,  mainly 
by  reports  from  industry  and  the  Defense  Laboratories.  This  value  addresses  the  call  to 
develop  training  capabilities  and  regimens  before  a  system  is  fielded,  as  advocated  in  the 
Strategic  Plan  for  the  Next  Generation  Training.  In  view  of  the  goals  of  the  study  and 
the  limitations  of  source  data,  a  gap  analysis  was  chosen  as  the  appropriate  research 
methodology. 

Gap  analysis  generally  refers  to  the  activity  of  studying  the  differences  between 
an  "existing  status'  and  a  "potential  status'  or  what  might  be  the  desired  state.  It  is  a 
method  more  commonly  used  in  the  study  of  business  processes  and  services,  such  as 
closing  the  gap  in  the  level  of  service  quality  or  the  time  for  service  delivery.  In  the 
context  of  the  Training  Systems  Study,  the  existing  status  is  the  documented  approach  to 
training  as  part  of  a  major  systems  acquisition,  especially  at  the  time  a  system  acquisition 
passes  Milestone  B,  the  entrance  of  the  System  Development  and  Demonstration  phase 
within  the  Defense  Acquisition  Management  Framework.  Analysis  of  these  data  and 
trends  would  address  the  two  central  research  questions.  The  "potential  status  ’  refers  to  a 
training  capability,  also  documented  in  the  literature,  for  the  same  or  a  like-kind  system 
that  has  demonstrated  level  of  effectiveness  that  might  exceed  that  of  the  training  system 
portrayed  in  Milestone  B  documentation.  Such  cases  of  exceptional  training  may  be 
considered  a  best  practice.  For  example,  there  may  be  documented  findings  concerning 
the  training  for  a  command  and  control  system  for  an  air-ground  system  in  the  land  forces 
that  could  equally  apply  to  training  for  a  maritime  command  and  control  system.  Gap 
analysis  relies  upon  determining  the  existing  status  is  for  a  system  and  comparing  it  to  a 
potential  status  at  a  particular  point  in  time. 

The  focus  of  the  early  work  on  the  study  is  to  create  a  comprehensive  record  of 
findings,  first  on  the  status  of  training  plans  and  intentions  for  an  existing  system  and 
secondly  on  the  status  of  training  capabilities  known  beforehand  or  at  the  same  point  in 
time.  To  accomplish  this,  source  materials  need  to  be  identified,  variables  need 
definition,  rules  for  coding  the  variables  need  explanation,  and  the  overall  record  needs  a 
database  structure  suitable  for  detailed  analysis.  The  four  key  steps  needed  to  be  taken 
are: 

1.  Seek  an  understanding  of  the  environment  surrounding  the  training  issue. 

2.  Take  a  holistic  view  of  the  acquisition  and  training  environment  to  gain  a 
complete  understanding. 

3.  Determine  what  framework  the  study  will  use  for  assessment  of  the  problem. 

4.  Provide  data  supporting  the  analysis  that  is  undertaken. 
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A.  STEP  1  -  UNDERSTANDING  THE  TRAINING  ISSUE 

The  systems  acquisition  process  is  governed  by  rules,  regulations,  instructions, 
handbooks,  etc.  Planning  documents  for  training  define  the  design,  development, 
funding,  equipment,  support,  modification,  operation,  and  management  for  future  training 
system  requirements.  The  training  system  design  typically  uses  the  instructional  systems 
development  (ISD)  process  to  determine  specific  training  system  characteristics, 
including  media,  training  requirements,  and  instructional  courses.  How  all  of  this  fits 
into  the  total  acquisition  picture  needs  to  be  understood,  and  how  it  has  changed  over 
time  is  important  to  recognize. 

B.  STEP  2  -  HOLISTIC  VIEW  OF  THE  ACQUISITION  AND 

TRAINING  ENVIRONMENT 

The  acquisition  of  a  major  system  is  a  long  and  complex  process  in  which  costs 
must  be  controlled  while  tradeoffs  are  being  made.  The  GAO  has  consistently  designated 
the  DoD  management  of  its  major  weapon  acquisitions  as  a  high-risk  area  (GAO-09- 
43  IT),  with  an  average  schedule  delay  in  delivering  initial  capabilities  of  21  months.  A 
perspective  is  needed  on  how  training  system  plans,  or  a  lack  thereof,  may  contribute  to 
system  deficiencies  and  delays,  while  considering  technological  advances  that  could 
improve  training  effectiveness/efficiency. 

C.  STEP  3  -  GAP  ANALYSIS  FRAMEWORK 

Gap  analysis  is  an  effective  solution  for  ongoing  performance  improvement  and 
quality  assurance  activities  that  identify  shortcomings  in  a  business  practice,  here  the 
acquisition  process.  Gap  analysis  utilizes  comparative  analysis  techniques,  thus  it  is 
essential  to  identify  the  nature  of  the  current  processes  in  order  to  make  comparisons. 
The  types  of  gaps  that  can  be  examined  are: 

Knowledge  gap  -  a  best  training  practice  has  not  been  fully  validated  or  proven 
for  application  for  a  given  training  system. 

Awareness  gap  -  a  best  training  practice  is  proven,  established,  and  relevant,  but 
has  not  been  applied  for  the  training  system  of  interest. 

Implementation  gap  -  a  valid  best  training  practice  has  been  identified  and 
attempted,  but  did  not  work  properly  in  the  case  of  a  given  training  system. 

Commitment  gap  -  a  valid  best  training  practice  is  recognized  but  not  applied  due 
to  policy,  cost,  schedule  or  other  factors. 

D.  STEP  4  -  DATA  FOR  ANALYSIS 

The  collection  of  valid  data,  based  to  the  greatest  extent  possible  on  empirical  findings 
and  from  trusted  sources  is  critical.  Not  acceptable  are  ‘sea  stories’,  anecdotal  evidence, 
or  other  forms  of  judgments  that  are  not  firmly  rooted  in  fact.  Benchmarking,  the  process 
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used  to  evaluate  performance  against  other  systems’  practices  and  performance  standards, 
is  also  an  acceptable  tool. 


E.  APPROACH 

The  main  approach  is  to  refine  the  concepts  into  a  workable  set  and  then 
formulate  a  set  of  variables  that  reflect  the  concepts  for  inclusion  in  the  database 
structure.  The  database  consists  of  an  organized  collection  of  data  captured  from  source 
documents.  Each  record  in  the  database  represents  an  individual  document,  although  it  is 
possible  that  multiple  documents  may  be  merged  into  a  single  record,  such  as  data  from  a 
GAO  report  on  a  particular  system  which  has  other  data  already  recorded.  As  depicted  in 
Table  1,  the  record  is  the  title  of  the  report,  represented  as  a  row  (in  SQL  terminology)  or 
a  tuple  (in  relational  terminology).  The  variables,  represented  as  a  column  (in  SQL 
terminology)  or  attribute  (in  relational  terminology)  represent  data  values.  Lor  example, 
a  Year  variable  indicates  the  year  of  publication  for  a  particular  report,  such  as  1999.  The 
data  types  will  vary,  and  coding  rules  will  specify  the  rules  and  constraints  for  variable 
declarations. 


Table  1.  Organization  of  Database 


Record  1 

Record  2 

etc. 

Variable  1 

Variable  2 

etc. 

The  study  team  then  considered  potential  variables.  The  principal  investigator 
proposed  a  starting  set  of  variables:  six  as  report  descriptors,  thirteen  related  to  system 
specifics,  and  seven  related  to  training  practices. 

F.  REFINEMENT  OF  VARIABLES 

The  study  team  reviewed  60  variables  nominated  by  individual  members  of  the 
team.  To  make  the  study  practical,  the  number  needed  to  be  reduced  by  about  one-third 
or  more.  Upon  feedback  from  the  sponsor  and  another  round  of  internal  review,  the 
variables  were  pared  down  based  on  discussion  and  consensus  building.  The  coding  rules 
for  the  database  were  then  articulated,  and  the  pursuit  of  reports  and  other  forms  of 
documentation  can  begin. 

The  variables  can  further  be  clustered  into  topical  areas  such  as  those  relating  to 
the  nature  of  the  report,  those  providing  specifics  on  the  system  and  acquisition  of 
training  for  the  system,  and  those  related  to  training  capabilities  from  experimentation 
with  the  system  or  with  like -kind  systems.  This  list  was  narrowed  several  times,  resulting 
in  37  variables  including  a  summary  comment  block.  The  final  set  of  variables  selected 
are  listed  in  Table  2.  The  coding  rules  are  presented  in  Appendix  B. 
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Table  2.  Listing  of  variables 

Variables 

Database  Entry  # 

Rater  Initials 

1 .  Report  Document  Title 

2.  ADA  number  (DTIC) 

3.  Other  reference  (journal,  book  chapter,  conference  paper,  other  report) 

4.  Pages 

5.  Corporate  Author 

6.  Year  of  Report 

7.  Type  of  Activity  publishing/submitting  report  (not  necessarily  sponsor) 

8.  Funding  for  Report  Document  (possible  R&D  funding  for  report  on  Major  System) 

9.  System  Name  or  Program  Number  (PNO)  for  more  recent  acquisitions 

10.  System  Type  -  Functional  Capability 

1 1 .  Acquisition  Phase  (as  of  Year  of  Report  variable) 

12.  Acquisition  Cost  -  Major  Defense  Acquisition  Program 

13.  Requirement  Source 

14.  Who  developed  training  -  prime,  sub 

15.  Training  Support  Plan  -  completeness,  depth,  etc. 

16.  System  Training  Plan  -  completeness,  depth,  etc. 

17.  Embedded  Training 

18.  System  Safety/Health  Hazards/Survivability  -  training  relationship  specified 

19.  Domain  Integration  -  training  tradeoffs  between  domains 

20.  Rapid  Acquisition  -  training  consideration 

21.  Spiral  Development  (or  incremental  analysis,  evolutionary  acquisition) 

22.  System  of  Systems  approach  -  training  consideration 

23.  Training  Effectiveness  Analysis  or  TE  Evaluation 

24.  Individual  Training  MOE/MOPs 

25.  Collective  MOE/MOPs 

26.  Maintenance  MOE/MOPs  (crew  related  &  system  related) 

27.  Operational  Test  Sites 
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28.  Based  on  empirical  evidence 

29.  Use  of  military  test  subjects 

30.  Quantifiable  Level  of  Learning  (Kirkpatrick  Model,  (Kirkpatrick,  1988)) 

31.  Based  on  multiple  studies 

32.  Proprietary  solution  for  training  delivery  (relates  to  S9) 

33.  Generalizability  within/across  system  types 

34.  Year  of  GAO  (or  DOTE,  IG)  report 

35.  Year  of  GAO/DOTE/IG  Report 

36.  Shortcomings  on  training  (effectiveness,  cost,  currency  etc.) 

37.  Summary  comments,  potential  gaps,  observations 

G.  SOURCE  DOCUMENTS 

Databases  for  a  computer-based  literature  searches  were  identified  early.  The 
primary  sources  are  the  Defense  Technical  Information  Center,  the  National  Simulation 
and  Training  Asssociation’s  collection  of  papers  from  the  EITSEC  conferences, 
Psychinfo,  and  the  ERIC  databases,  and  also  collections  at  the  Naval  Postgraduate  School 
and  the  Air  Eorce  Institute  of  Technology.  Inquiries  to  relevant  programs  and  system 
documentation  in  the  Services,  the  Joint  Staff,  and  the  Defense  Acquisition  University  led 
to  additional  sources.  The  reports  from  these  sources  belong  to  the  general  category  of 
Technical  Reports.  The  annual  report  of  the  Director,  Operational  Test  and  Evaluation, 
reports  from  the  DoD  Inspector  General,  and  reports  from  the  General  Accountability 
Office  served  as  sources  for  documented  shortcomings  in  acquisitions  from  a  training 
standpoint.  The  sources  belong  to  the  general  category  of  Consequential  Reports. 

The  study  team  then  focused  on  source  documents.  Some  initial  results  of  searching 
data  sources  include  2,694  hits  on  the  Defense  Technical  Information  Center  (DTIC) 
database  using  the  terms  “training  AND  acquisition.”  A  search  of  the  General 
Accountability  Office  (GAO)  database  resulted  in  1,600  hits  using  the  term  “Defense 
Acquisition.”  A  secondary  search  of  a  small  number  of  individual  reports  returned 
occasional  references  to  training,  some  incidental  some  useful.  Eor  example,  an  examination 
of  a  1985  GAO  report  (GAO/NSIAD-85-154)  to  the  Secretary  of  the  Army  on  integrating 
manpower,  personnel  and  training  into  the  weapons  acquisition  process  contained  an 
appendix  listing  of  85  reports  on  the  topic  from  across  the  Services  and  DoD.  Other  sources 
identified  are  two  academic  databases,  IITSEC  papers,  theses  and  dissertations,  reports 
available  from  the  Defense  Acquisition  University,  documentation  from  program 
management  offices,  reports  from  Operational  Testing  and  Evaluation,  and  internal  industry 
reports.  MANPRINT  assessments  and  related  reports  from  the  Army  were  also  sampled 
from  the  comprehensive  collection  assembled  by  the  U.S.  Army  Research  Eaboratory 
(Middlebrooks,  2010).  Cost  data  were  derived  primarily  from  the  Selected  Acquisition 
Report  Summary  Tables,  provided  every  month  by  Office  of  the  Under  Secretary  for 
Defense,  Acquisition  Technology  &  Eogistics.  The  data  for  the  month  of  June  were  used  for 
a  given  year.  The  net  search  would  likely  lead  to  several  thousand  source  documents,  a 
selected  set  which  would  be  coded  for  Phase  I  of  the  study. 
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Figure  1  depicts  the  flow  of  materials  from  the  two  categories  of  source  materials  to 
the  coded  database. 


Coded  Database 


Figure  1 .  Depiction  of  Coding  Process 

H.  CODING  DATABASE 

Searches  on  key  databases  resulted  in  more  than  4,000  candidate  reports  that  were 
technical  in  nature.  Since  the  nature  of  the  study  was  exploratory,  the  list  was  narrowed. 
Each  report  was  searched  for  the  term  ‘train’  and  if  a  rater  judged  the  context  as  relevant 
to  the  study,  a  copy  of  the  report  was  obtained  and  coded.  Members  of  the  study  team 
served  as  coders.  A  workshop  in  March  2010  provided  examples  of  coding  with 
feedback  and  comments  by  team  members  as  a  means  to  reach  agreements  in  coding 
protocol  and  later  maintain  a  high  level  of  coding  conformity.  Many  of  the  variables, 
though,  were  simply  facts  identified  in  the  study  reports  or  from  external  sources  on  a 
particular  acquisition  program,  such  as  costs. 

Abstractions  of  the  database  are  provided  in  Appendix  C,  with  separate  sections 
for  the  technical  and  GAO  reports.  The  technical  reports  are  presented  in  chronological 
order  so  the  database  entry  is  a  unique  but  arbitrary  assignment  used  for  quickly  locating 
the  record.  For  the  purposes  of  the  immediate  gap  analysis  presented  later  in  this  report, 
the  abstractions  provide:  database  entry  (locator),  title  of  report,  year  of  report.  Service  or 
Joint,  name  of  system  or  training  device,  whether  human  performance  data  were  reported. 
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whether  the  findings  can  be  generalized  to  other  systems,  and  a  brief  description  of  the 
finding  as  it  relates  to  the  goals  of  the  study.  An  example  is: 

49 

Human  Factors  Performance  Data  for  Future  Forward  Area  Air  Defense  Systems  (FAADS) 

1989 

Army 

Forward  Area  Air  Defense  Systems  (FAADS) 

Human  performance  data  reported 

Generalizable 

Testifying  before  Congress  in  1977,  an  Army  official  stated  that  "We  expect  this  somewhat  unorthodox  approach  to 
permit  a  much  reduced  development  time,  thus  resulting  in  an  earliest  fielding  date,  albeit  with  higher  but  acceptable 
risks.”  Despite  the  use  of  many  off  the  shelf  technologies  that  were  intended  to  allow  rapid  and  low-cost 
development,  a  series  of  technical  problems  and  massive  cost  overruns  resulted  in  the  cancellation  of  the  Sergeant 
York  project  in  1985  by  the  Secretary  of  Defense.  Although  much  of  the  processing  of  the  collected  data  was  also 
cancelled  or  brought  to  a  hasty  conclusion,  an  extensive  follow-on  evaluation  was  conducted.  The  research 
documented  the  analysis  of  training  and  human  factors -related  data  from  the  Sgt  York  Follow-On  Evaluation  I,  which 
generated  objective  performance  measures  on  individual  soldier,  team,  and  system  performance.  Objective 
performance  data  were  examined  with  respect  to  fourteen  issues,  such  as  varying  target  workloads  on  the  operator  or 
the  negative  impact  of  some  semi-automation.  The  data  yielded  findings  on  many  issues  relevant  to  the  design  of  a 
future  area  air  defense  system.  The  report  provides  suggestions  on  applying  such  data  to  future  soldier-system 
performance  models,  which  can  be  remarkably  helpful  in  making  decisions  about  future  FAADS.  Awareness  Gap 
input:  Failures  of  cancelled  system  can  lead  to  training  and  performance  improvements  in  similar,  follow  on  systems 
if  properly  analyzed.  Implementation  Gap  input:  Maintaining  detailed  human  performance  data  during  DT/OT  allows 
secondary  analyses  for  other  system  designers. 

The  GAO  reports  were  abstracted  in  a  slightly  different  manner.  To  distinguish 
these  from  the  technical  reports,  the  database  entry  numbering  begins  with  500.  The 
abstractions  provide:  database  entry  (locator),  title,  year.  Service  or  Joint,  system,  GAO 
report  identification,  passabe  from  report  with  context  for  training  issue.  An  example  is: 

503 

Missile  Risk  Reduction  Underway  But  System  Production  Plans  Need  to  be  Reexamined 

1998 

Joint 

AIM-9X,  Sidewinder 
NSIAD-98-45 

In  commenting  on  a  draft  of  this  report,  DOD  stated  that  the  projected  range  and  sensor  tracking  capability  of  AIM-9X 
without  the  helmet-mounted  cueing  system  is  equivalent  to  the  capability  of  the  AA-11  threat  missile  in  azimuth  and 
exceeds  the  capability  of  the  AA-11  in  range.  DOD’s  position  is  based  on  using  the  fighter  aircraft  radar  to  cue  the 
AIM-9X  missile  to  the  target  of  interest  when  it  is  beyond  the  view  of  the  aircraft’s  heads-up  display.  Using  the  radar 
to  cue  the  missile,  however,  will  take  more  time  and  be  less  certain  than  with  the  helmet  and  will  require  DOD  to 
train  pilots  in  yet  to  be  developed  procedures  and  tactics  that  would  be  considerably  different  than  current  practices 
for  aerial  combat. 
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VI.  RESULTS 


A.  DESCRIPTIVE  STATISTICS 

Technical  reports  were  drawn  from  1970  to  2011.  By  decade,  the  frequency  of 
reports  is  presented  in  Table  3.  There  was  a  relatively  even  distribution  of  the  publication 
dates  across  the  four  decades  which  we  examined. 

Table  3.  Distribution  of  technical  reports  by  decade 


Decade 

Percentage  of  Reports 

1970-  1979 

25% 

1980-  1989 

30% 

1990-  1999 

23% 

2000-2011 

22% 

The  distribution  of  reports  by  Service  is  presented  in  Table  4.  The  reports  were 
approximately  evenly  distributed.  Since  the  Navy  and  Marine  Corps  have  a  common 
acquisition  authority,  they  were  aggregated. 

Table  4.  Distribution  of  technical  reports  by  decade 


Service 

Percentage  of 
Reports 

Army 

37% 

Navy  &  Marine  Corps 

33% 

Air  Force 

30% 

The  systems  that  were  examined  in  this  review  are  listed  in  Table  5,  along  with 
the  year(s)  of  the  report  in  which  the  system  was  examined.  Each  system  listed  below 
was  highlighted  in  the  report  with  a  specific  training-related  issue.  Thirty-one  systems 
were  identified. 
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Table  5.  Systems  reported  in  the  literature  searches 


System 

Year 

System 

Year 

A-7 

1976 

AIM-9X 

(Sidewinder) 

1998 

P-3C 

1977 

FBCB2 

1998 

T-37 

1978 

JCCS 

1998 

AC-130E 

1978 

MILSTAR 

1998 

CH-47 

1979 

Javelin 

1998,  1999 

AH-1 

1980 

F-22 

1994,  1999,  2004 

H-3,  SH-3 

1981 

V-22 

2001 

F-111 

1982 

KC-135 

2004,  2010 

C-130 

1982 

JSF 

2004 

F-16 

1987 

LCS 

2004 

FAADS 

1990 

Longbow 

2006 

B-2 

1993,  1997 

Patriot 

2007 

AH-64 

1994 

Predator 

2009 

JSTARS 

1996 

Global  Hawk 

2009 

THAAD 

1996,  1999 

Reaper 

2009 

JPATS 

1997,  2003 

B.  ANALYSIS  OF  GAPS 

In  a  speech  that  was  in  the  Congressional  Record  dated  October  1,  1997, 
Representative  Ike  Skelton  shared  with  his  colleagues  his  views  on  the  successes  of  the 
MANPRINT  program.  He  pointed  to  notable  MANPRINT  assessments  that  led  to 
operational  improvements,  such  as  the  reduction  in  tools  for  a  Comanche  helicopter  T- 
800  engine  from  134  to  only  6.  In  an  example  in  which  MANPRINT  considerations  were 
not  incorporated  and  in  fact  purposely  rejected,  according  to  the  Army  Audit  Agency,  the 
Armored  Gun  System  was  cancelled  prior  to  its  being  fielded  for  several  reasons 
including  HSI  factors.  Booher  (1990)  provides  additional  examples  from  the  1980s. 
Such  problems  occurred  when  human  performance  assessments  were  either  not  done  or 
done  too  late  to  have  an  impact  on  the  system  design  process.  Representative  Skelton 
argued  that  MANPRINT  domains,  of  which  training  is  one,  were  not  being  integrated  in 
the  acquisition  process  on  a  continuous  basis.  There  were  gaps  between  a  hoped-for 
potential  status  versus  an  actual  existing  status  at  critical  times  in  the  acquisition  cycle. 

The  gap  analysis  in  this  study  examines  differences  in  the  existing  status  of 
training  for  a  system  as  reflected  in  a  published  report  and  the  potential  of  what  could 
have  been  the  desired  state  if  training  plans  or  training  execution  had  been  done  more 
efficiently  or  effectively.  It  is  easy,  certainly,  to  look  back  at  a  consequential  fault  and 
recommend  an  “if  only  they  had  done  this  or  considered  that.”  We  cannot,  of  course, 
judge  with  certainty  if  alternatives  were  considered  or  plans  were  reduced  for  a  variety  of 
reasons,  many  of  them  may  have  seemed  reasonable  at  the  time.  What  we  can  do  is 
detect  and  classify  a  pattern  of  gaps  that  highlight  the  importance  of  early  design  of 
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training  and  well-planned  training  programs  and  what  the  consequences  can  be  if  training 
is  not  fully  integrated  throughout  the  acquisition.  There  is  a  degree  of  supposition  on  our 
part  as  to  the  causes  of  why  these  gaps  materialized,  so  we  can  go  no  further  than  offering 
a  professional  appraisal  based  on  our  working  experience  in  the  field.  That  these  gaps 
persist  is  reason  to  explore  the  problem  area,  and  seek  relief  through  policy  change  and 
educating  the  acquisition  and  industrial  workforces.  Future  work  can  consider 
extensions,  deeper  fact-finding  approaches,  or  case  study  methods  to  understand  the  issue 
and  its  antecedent  causes  in  greater  depth  and  with  greater  accuracy. 

Four  categories  of  gaps  were  identified  earlier.  For  each  gap,  we  supply  instances 
and  examples  drawn  from  the  database.  This  example  serves  to  suggest  this 
methodology  as  a  way  to  explore  further  questions  surrounding  the  value  of  training  early 
in  an  acquisition  program.  The  results  are  organized  by  the  type  of  gap,  which  was 
determined  based  on  a  comparison  of  a  training  consequence  from  a  GAO  record  with 
one  or  more  findings  in  the  technical  database,  a  summary  statement  of  the  consequence, 
and  a  record  number.  The  record  number  refers  to  the  database  locator.  A  listing  of 
database  entries  and  abstracts  is  provided  in  Appendix  C,  with  the  technical  entries  listed 
chronologically  and  the  GAO  entries  listed  in  order  of  entry. 

Knowledge  Gap  -  a  best  training  practice  has  not  been  fully  validated  or  proven 
for  application  for  a  given  training  system 

•  Training  in  a  Navy  aviation  logistics  system  was  augmented  by  word-of-mouth 
training  due  to  a  lack  of  proper  technical  documentation  and  availability  of 
manuals,  raising  costs  and  resulting  in  an  instance  of  a  knowledge  gap  (record 
516,  1990). 

•  Insufficient  estimates  for  supporting  costs  related  to  training,  particularly 
maintenance  training,  appears  to  have  resulted  from  a  lacking  front  end  analysis, 
resulting  in  a  knowledge  gap  (record  506,  1993). 

•  Pilot  training  for  the  Predator  advanced  concept  technology  demonstration  was 
found  to  be  deficient  since  the  demonstration  was  not  designed  to  do  so,  an 
example  of  a  knowledge  gap  (record  504,  1998). 

Awareness  Gap  -  a  best  training  practice  is  a  proven  and  established  method,  but 
has  not  been  applied  for  the  training  system  of  interest. 

•  Insufficient  front  end  analysis  for  special  operations  aircrew  training  led  to  a 
schedule  slippage,  resulting  in  an  instance  of  an  awareness  gap  (record  519, 

1992). 

•  Training  targeting  using  the  radar  to  cue  a  missile,  rather  than  using  a  helmet- 
mounted  cueing  system,  requires  training  in  procedures  and  tactics  that  have  not 
yet  been  developed,  despite  training  methods  capable  of  delivering  such 
instruction,  resulting  in  an  awareness  gap  (record  503,  1998). 
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•  Operator  training  for  a  command  and  control  system  was  found  to  be  inadequate, 
despite  successes  with  training  such  skills  in  the  technology  base,  resulting  in  an 
awareness  gap  (record  510,  1998). 

•  The  training  of  gunners  for  the  Javelin  antitank  weapon  limits  the  weapons 
capabilities,  notably  due  to  training  software  that  will  not  teach  gunners  to 
recognize  moving  targets  or  properly  train  identification  and  recognition  of 
thermal  images,  resulting  in  an  awareness  gap  (record  511,  1998). 

•  Training  shortfall  led  to  shipboard  operators  performing  manual  tuning  of  radio 
frequencies  without  prior  training  due  to  reliability  issues,  resulting  in  an 
awareness  gap  (record  527,  2001). 

•  Lack  of  synchronization  between  needed  training  capabilities,  particularly  the 
information  management  of  training,  and  production  decisions  in  primary  aircraft 
training  system  resulted  in  an  awareness  gap  (record  530,  2003). 

•  The  availability  of  F/A-22  and  Joint  Strike  Fighter  aircraft  for  training  (or 
combat)  use  can  be  diminished  without  improvements  in  reliability,  resulting  in 
an  awareness  gap  (record  532,  2004). 

•  Inadequate  front  end  analysis  reduced  the  availability  of  refueling  aircraft  for  the 
Navy  due  to  a  lower  priority  placed  by  the  Air  Force,  resulting  in  an  awareness 
gap  (record  533,  2004). 

•  A  weak  cost  estimate  for  training  due  in  part  to  new  operational  concepts  for 
multi-skilled  sailors  to  operate  and  maintain  the  Littoral  Combat  Ship  resulted  in 
an  awareness  gap  (record  535,  2010). 

Implementation  Gap  -  a  valid  best  training  practice  has  been  identified  and 
attempted,  but  did  not  work  in  the  case  of  a  given  training  system. 

•  Aircraft  are  not  fully  available  for  crew  training,  so  not  planning  sufficient 
aircraft,  simulators,  or  part  task  trainers  for  training  purposes  through  better  front- 
end  planning  limits  operational  effectiveness,  resulting  in  an  instance  of  an 
implementation  gap  (record  501,  1997). 

•  Training  requirements  may  be  able  to  be  met  with  a  smaller  quantity  of  primary 
aircraft  training  system,  which  appears  not  to  be  following  a  proper  economical 
approach  during  early  analysis,  resulting  in  an  implementation  gap  (record  505, 
1997). 

•  Inadequate  collective  training  for  units  contributed  to  an  inadequate  experiment 
related  to  operational  testing  of  battle  command  system,  which  may  have  resulted 
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from  an  inadequate  front  end  analysis  of  levels-of-training  needed  for  testing, 
resulting  in  an  implementation  gap  (record  508,  1998). 

•  Changes  resulting  from  upgrades  to  communication  satellites  required  a 
reconfiguration  of  software  and  a  retraining  of  operators;  a  more  inclusive  front 
end  analysis  could  have  better  modularized  the  training  and  reduced  the  retrain 
requirements,  resulting  in  an  implementation  gap  (record  514,  1998). 

•  An  adjustment  in  the  number  and  model  type  of  end  items  to  be  acquired  (fighter 
jets)  resulted  in  a  greater  percentage  of  trainer  aircraft,  with  less  capability,  as  part 
of  the  total  buy.  Expanding  the  use  of  simulation  and  simulators  may  have  been 
able  to  mitigate  this,  resulting  in  an  implementation  gap  (record  509,  1999). 

•  A  change  to  the  Javelin  system’s  launch  unit  resulted  in  requiring  personnel  to  be 
trained  for  two  configurations,  an  example  of  training  not  keeping  up  with 
engineering  changes  in  the  system,  resulting  in  an  implementation  gap  (record 
512,  1999). 

•  Inadequate  training  of  shipboard  technicians  resulted  in  an  implementation  gap 
(record  524,  2000). 

•  Inadequate  front  end  analysis  in  several  tactical  aircraft  acquisitions  translates  to 
fewer  aircraft  being  available  for  training  (record  529,  2001). 

•  Win-T  (record  534,  2011). 

Commitment  Gap  -  a  valid  best  training  practice  is  recognized  but  not  applied 
due  to  policy,  cost,  schedule  or  other  factors. 

•  A  general  issue  raised  was  a  practice  of  program  sponsors  maintaining  low  cost 
estimates  by  excluding  costs  such  as  costs  of  training  equipment,  resulting  in  an 
instance  of  a  commitment  gap  (record  517,  record  521  1992). 

•  Shortfalls  in  operator  training  for  the  Joint  Stars  Ground  Station  caused  problems 
during  operational  testing,  many  later  isolated  to  software  fixes  resulting  in  a 
commitment  gap  (record  513,  1996). 

•  Additional  training  is  needed  to  improve  anti  submarine  warfare  crew  proficiency, 
so  training  resources  were  insufficient  or  not  properly  committed  in  view  of  other 
priorities,  resulting  in  a  commitment  gap  (record  502,  1999). 

•  Potential  over  reliance  on  contractor  personnel  in  general  may  reduce  rotational 
positions  to  meet  training  requirements  in  logistics  support  areas,  resulting  in  a 
commitment  gap  (record  525,  2000). 
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•  In  February  2010  the  program  reported  a  Nunn-McCurdy  unit  cost  breach  of  the 
significant  cost  growth  threshold,  which  it  attributed  to  factors  such  as  the 
omission  of  training  devices  and  adequate  spares  from  initial  estimates,  and 
delays  in  the  production  decision,  (record  534,  2011). 

Twenty  six  gaps  were  uncovered  based  on  judgments  of  training  analysts 
involved  in  the  study.  Undoubtedly,  many  more  could  be  identified  with  a  more 
thorough  analysis  and  other  source  materials  from  operational  testing  results.  The 
breakout  of  type  of  gap  by  category  is  presented  in  Table  6. 

Table  6.  Percentage  of  cases  for  each  of  the  gaps  studied 


Percentage 

Knowledge  Gap 

11% 

Awareness  Gap 

35% 

Implementation  Gap 

35% 

Commitment  Gap 

19% 

C.  CROSSWALK  WITH  TECHNICAL  REPORT  DATABASE 

The  thrust  of  this  analysis  is  to  determine  the  extent  to  which  there  was  pre¬ 
existing  knowledge  in  the  technology  base  that  could  have  potentially  alleviated  or  even 
eliminated  the  gap.  In  each  case  of  the  gaps  identified,  information  in  the  Technical 
Report  portion  of  the  database  indicated  the  availability  of  one  or  more  documents  that 
preceded  the  documented  circumstance  and  could  have  potentially  circumvented  the  gap, 
at  least  in  a  notional  way  without  deeper  study  of  the  details.  For  example,  the  awareness 
gap  identified  as  insufficient  estimates  for  supporting  costs  related  to  training, 
particularly  maintenance  training,  appears  to  have  resulted  from  a  lacking  front  end 
analysis  (record  506,  1993)  could  have  potentially  been  remediated  by  applying  the 
knowledge  gained  in  an  Air  Force  report,  or  follow  on  work,  describing  a  methodology 
which  is  useful  for  applying  human  resources,  logistics,  and  cost  factors  in  weapon 
system  acquisition  programs.  The  methodology,  termed  the  coordinated  human  resources 
technology  (CHRT),  was  developed  from  an  integration  of  five  individual  human 
resources  technologies:  maintenance  manpower  modeling,  instructional  system 
development,  job  guide  development,  human  resources  in  design  tradeoffs,  and  system 
ownership  costing  (record  30,  1980). 

D.  FRONT  END  ANALYSIS  AND  GAPS 

The  Defense  acquisition  process  consists  of  the  phases  and  milestones 
described  in  earlier  paragraphs.  During  the  early  or  “front  end”  of  the  acquisition 
process,  the  Concept  Refinement  Phase,  incorporates  detailed  analysis  including  an 
analysis  of  alternatives  to  assess  various  materiel,  software,  and  technologies  solutions  to 
shape  capabilities  to  best  meet  the  military  operational  requirements.  A  recent  topic  of 
discussion  in  Defense  Leadership  meetings  has  been  the  need  for  more  rigorous 
assessments  in  the  earliest  stages  of  the  acquisition  process  can  be  most  productive  in 
effectively  and  efficiently  meeting  the  warfighter  needs.  It  is  further  believed  that  early 
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incorporation  of  system  training  details  in  the  pre-milestone  A  analysis  has  great  benefit 
in  achieving  the  best  possible  system  solution  and  can  positively  impact  each  stage  of  the 
acquisition  life  cycle.  Particularly  those  systems  with  significant  human  interface 
requirements  are  found  to  benefit  from  bringing  in  training  systems  planning  in  the 
concept  refinement  process  in  a  series  of  iterations  with  the  end  users.  Early  systems 
prototypes  (constructive  or  virtual  simulations)  for  instance  can  be  a  very  effective  tool  in 
narrowing  down  and  shaping  the  final  desired  capability  with  the  warfighters.  The  use  of 
modeling  and  simulation  and  incorporation  of  a  systems  engineering  approach  to  systems 
training  can  logically  drive  the  future  system  training  tools,  simulators  and  part-task 
trainers,  as  appropriate.  In  major  acquisition  programs  the  ability  to  insert  training  early 
in  the  program  can  help  shape  the  final  quantities  and  technical  complexity  of  the  end 
capability.  From  a  senior  DoD  management  perspective,  proper  front  end  analysis  to 
include  the  systems  training  considerations  can  permit  a  go  or  no-go  decision  and/or 
justify  significant  modifications  prior  to  the  commitment  of  large,  long-term  fiscal 
resources. 

VIL  DISCUSSION  AND  CONCLUSION 

Representative  Skelton,  in  his  speech  to  Congress  referenced  earlier,  stated  ‘it 
seemed  that  whenever  a  new  system  was  put  into  the  hands  of  the  soldier,  actual  field 
performance  often  failed  to  match  the  standards  predicted  during  its  development.’  What 
went  wrong  with  the  acquisition?  This  report  defined  a  potential  acquisition  concern  - 
slow  or  poor  integration  of  training  during  weapon  system  development,  proposed 
methodologies  needed  to  examine  further  issues  surrounding  training  system  integration. 
The  study  reviewed  selected  documents  that  reveal  that  the  direction  of  training  R&D,  the 
status  of  training  practices,  and  shortcomings  in  systems  being  acquired  by  DoD  running 
from  the  early  1970s  to  early  2011.  The  review  of  technical  reports  focused  primarily  on 
determining  the  training  effectiveness  of  specific  training  systems  as  related  to 
technological  and  methodological  concerns,  while  the  other  reports  highlighted  training- 
related  issues  identified  primarily  during  developmental  and  operational  testing. 

There  were  thousands  of  reports  to  choose  from,  so  a  systematic  sample  was 
selected  from  the  technical  side  given  the  resource  limitations  of  the  study.  The  reports 
related  to  training  issues  for  specific  systems  were  more  complete  over  a  twenty  year 
span.  Given  the  particular  sample  of  reports  reviewed  here,  we  cannot  fully  judge  the 
impact  of  late,  or  inadequate  training  support  for  a  given  weapon  system,  nor  can  we 
make  the  general  case  relating  training  development  timing  and  weapon  system  mission 
readiness  and  operational  effectives.  However,  we  suspect  that  further  inquiry  into  the 
subject,  with  additional  source  materials  and  the  inclusion  of  complementary  methods 
such  as  case  study  and  archival  document  content  analysis  would  be  productive  and 
bringing  us  closer  to  answering  the  stated  research  questions  regarding  the  impact  of 
early  training  system  development  on  weapon  system  acquisition  and  deployment. 

It  is  therefore  suggested  that  further  investigation  continue,  using  the  database 
formulated  during  this  study  as  a  point  of  departure.  Also,  it  is  believed  that  the  study 
methodology  should  consider  incorporating  systematic  case  study  analysis  (Yin,  1997)  to 
identify  program  outcomes  (successful  and  unsuccessful)  for  benchmarking  and 
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establishing  a  final  Lessons  Learned  Database.  A  transition  to  a  second  phase  is 
warranted. 

The  development  of  training  early  in  the  acquisition  can  have  other,  unexpected 
benehts.  For  example,  in  developing  the  Army’s  future  forward  air  defense  system, 
analysts  discovered  what  training  revealed  -  that  the  projected  air  threat  could  strike 
ground  forces  from  a  distance  beyond  the  range  of  the  missile  system.  The  Secretary  of 
Defense  cancelled  the  DIVAD  program  in  1985. 

Top  management  within  the  Department  of  Defense  has  taken  steps  in  an  attempt 
to  make  the  acquisition  process  more  disciplined,  and  it  has  redefined  the  basic  strategy 
for  acquiring  weapons.  For  example,  the  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  is  the  current  DoD  procedure  which  defines  acquisition  requirements 
and  evaluation  criteria.  It  replaces  the  previous  service-specific  requirements  systems, 
which  focused  on  addressing  future  threat  scenarios.  New  acquisition  programs  were  not 
considered  in  the  context  of  other  programs,  such  as  would  be  the  case  in  a  system-of- 
systems  approach.  These  previous  practices  created  redundancies  in  capabilities  and 
failed  to  meet  the  combined  needs  of  all  the  Services.  JCIDS  guides  the  development  of 
requirements  for  future  acquisition  systems  to  reflect  the  needs  of  ah  four  services.  The 
central  theme  is  to  address  capability  shortfalls,  or  gaps  as  defined  by  combatant 
commanders.  Thus,  JCIDS  is  said  to  provide  a  capabilities-based  approach  to 
requirements  generation. 

The  reform  of  the  acquisition  process  seeks  to  advance  technology  while  holding 
down  procurement  costs.  Better  training  of  the  acquisition  workforce,  tighter  monitoring 
of  contractor  performance,  and  the  availability  of  tools  for  those  involved  in  acquisition 
ah  contribute  to  reform  activities.  As  an  example  of  an  overarching  tool,  the  basic 
concepts  of  the  Simulation  Based  Acquisition  (SBA)  program  formed  a  decade  ago  are 
still  viable  -  but  largely  unachieved  today.  The  SBA  vision  encompassed  an  acquisition 
process  in  which  the  DoD  and  industry  partners  are  enabled  by  the  robust,  collaborative 
use  of  simulation  technology  integrated  across  ah  acquisition  phases  and  programs.  The 
goals  of  SBA  are  still  in  sync  and  consistent  with  the  current  acquisition  reform  policy 
initiatives. 

•  Reduce  time,  resources,  and  risk  associated  with  the  entire  acquisition  process 

•  Increase  the  quality,  military  worth  and  supportability  of  fielded  systems 

•  Reduce  total  ownership  costs  throughout  the  system  life  cycle,  and 

•  Enable  integrated  product  and  process  development  across  the  entire  acquisition 
life  cycle 

In  keeping  with  the  SBA  vision  and  goals,  the  Department  can  provide  a  systems 
engineering  environment  that  emphasizes  modeling  and  simulation  (M&S)  as  a  primary 
analysis  tool  and  fosters  the  use  and  reuse  of  data  and  M&S  content  across  programs  and 
phases.  It  is  envisioned  that  use  of  models  can  refine  the  needs  and  provide  the 
underpinning  for  more  rigorous  analyses  prior  to  Milestone  A,  while  transitioning  critical 
content  to  guide  systems  design  and  later  development  and  production  processes. 
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From  the  early  requirements  and  conceptualization  stages,  the  use  of  M&S  and  in 
particular  system  prototyping  provides  a  powerful  analytic  capability  to  meet  user  needs. 
It  has  been  argued  that  prototypes  are  platforms  for  productive  participation,  as  well  as 
for  perfecting  products  and  performance.  The  power  of  producing  systems  prototypes 
early  in  the  process  serves  as  a  way  to  iterate  feedback  from  and  dialogue  with  the  end 
user  to  arrive  at  better  systems  and  solutions  for  the  operational  needs.  The  more  obvious 
use  of  prototypes  is  to  guide  the  engineering  analysis  in  the  development  planning  stage 
of  the  acquisition.  Many  firms  can  be  found  proposing  services  to  industry  in  the  area  of 
model  building  and  prototyping  and  are  highly  successful  in  providing  rapid  prototyping 
services  that  encompass  proof  of  concept  and  proof  of  design  with  functional  working 
simulations  and  models.  The  use  of  prototyping  can  encompass  constructive  simulations, 
virtual  environments  or  physical  mock  ups  of  the  end  system  or  product.  With  the  use  of 
such  tools  as  3D  visualization  one  can  progress  to  “model  making”  to  influence  the 
construct  of  actual  3D  models.  The  area  of  rapid  prototyping  uses  state  of  the  art 
CAD/CAM  (computer-aided  design  and  computer-aided  machining  or  modeling) 
techniques.  Significant  advances  in  the  area  of  M&S  make  it  now  more  important  than 
ever  that  we  incorporate  oversight  directives  to  include  contracting  language  that  require 
the  use  of  simulations,  models  and  prototypes  in  all  phases  of  the  acquisition  process. 

Another  example  of  an  acquisition  workforce  tool  is  the  Improved  Performance  Research 
Integration  Tool,  or  IMPRINT.  Developed  by  the  U.S.  Army  Research  Laboratory, 
Human  Research  and  Engineering  Directorate,  IMPRINT  is  a  network  modeling  tools 
that  factors  in  people-driven  constraints  on  system  design,  evaluating  the  capability  of 
operators  and  maintainers  to  gain  optimal  performance  from  a  system  under 
environmental  stressors.  The  IMPRINT  Pro  tool,  available  free  of  charge  for  U.S. 
government  agencies  and  industry  under  contract  to  the  U.S  government,  now  has  the 
capability  to  examine  missions  and  systems  from  the  Services  and  Joint  commands. 

During  the  last  decade  the  Department  has  initiated  programs  such  as  “lean  six 
sigma”  and  “streamlining  acquisition”  to  make  the  acquisition  process  more  effective  as 
well  as  more  efficient.  Today  there  seems  to  be  a  growing  consensus  that  more  rigorous 
analysis  should  be  included  in  acquisition,  and  especially  in  the  early  phases  of  the 
process.  Consistent  with  these  approach  training  systems  considerations  should  be 
included  in  the  requirements,  concept  definition  and  refinement  phases  of  the  process. 
Early  in  his  administration.  President  Obama  announced  acquisition  reform  goals  and 
policies  and  outlined  actions  that  impact  government  procurement,  acquisition  programs 
and  contractors  in  a  wide  variety  of  areas.  The  convergence  of  new  administration 
priorities,  burgeoning  costs,  and  outdated  procurement  processes  prompted  a  series  of 
major  contracting  and  policy  initiatives  designed  to: 

•  Develop  more  agile  acquisition  processes  to  increase  the  speed  of  technology 
deployment 

•  Increase  transparency  of  the  acquisition  process 

•  Institute  stricter  risk  and  performance  parameters 
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•  Reduce  costs  through  cuts  in  contractor  spending  and  use  of  “high-risk” 
contract  types 

At  the  International  Training  Equipment  Conference  (ITEC)  held  May  10-12, 
2011  in  Cologne  German,  Director  of  Training  Readiness  and  Strategy  in  the  office  of  the 
Deputy  Assistant  Secretary  of  Defense  for  Readiness  again  expressed  the  belief,  from  the 
training  community,  that  acquisition  practices,  particularly  in  the  modeling  and 
simulation  area,  have  not  kept  pace  with  technology  advances.  He  went  on  to  say,  “We 
have  an  Industrial  Age  approach  to  acquisition,  and  we  are  in  an  e-commerce 
environment.  ”  In  today’s  e-commerce  environment  there  is  an  opportunity  to  track 
customer  preferences  and  look  at  demand  from  the  user  community  in  real  time.  As  part 
of  his  ITEC  presentation  the  Director,  TRS  appealed  to  the  military  to  adopt  more 
training  tools  that  can  customize  content  to  the  user  by  understanding  their  learning  style 
and  allowing  them  to  progress  at  a  more  natural  pace.  He  appealed  industry  to  embrace  a 
new  business  model  and  to  recognize  training  includes  the  soft  skills  found  in  today’s  full 
spectrum  operations  to  maximize  human  performance  potential. 


A.  CLOSING  COMMENTS 

The  DoD  spends  billions  of  dollars  each  year  developing  and  procuring  major 
weapon  systems.  These  expenditures  have  produced  the  most  technologically  advanced 
weapon  systems.  Nevertheless,  the  process  through  which  systems  are  determined  and 
acquired  has  often  proved  costly  and  inefficient,  as  hundreds  of  reports  by  the  DoD 
Inspector  General  and  the  Government  Accountability  Office  have  repeatedly  reported. 
Cost  and  schedule  can  be  overly  optimistic,  leading  to  instability  and  cost  increases  in  the 
long  run.  Powerful  incentives  and  interests  can  influence  the  behaviors  of  those 
involved,  including  DoD  components,  industry,  and  Congress.  More  funds  may  be  made 
available  for  training  and  other  human  system  considerations  through  a  reduction  of 
inefficiencies  without  necessarily  adding  to  baseline  costs.  Early  consideration  of 
training  can  have  clear  benefits  in  reducing  certain  problems  that  have  been  documented 
in  weapon  system  acquisition. 

The  Phase  I  effort  provides  a  relatively  extensive  list  of  reports  and  studies 
relating  to  training  systems  acquisition  in  MDAP  programs.  The  automated  data  base 
with  common  variables  provides  ready  access  to  details  from  several  hundred  program 
documents.  The  gap  and  trend  analysis  of  this  data  has  provided  a  start  point  for  the 
Phase  II  study  which  will  examine  a  series  of  selected  systems  (by  type  and  category)  for 
detailed  assessment  through  case  studies.  During  Phase  II  the  database  of  historic 
examples  will  be  extended,  as  appropriate,  to  flesh  out  the  selected  case  studies.  Eor  each 
system  chosen,  specific  metric  will  be  developed  to  assess  the  early  and  effective  system 
training  practices  in  acquisition.  The  selection  of  both  common  and  system  specific 
metrics  will  allow  the  research  team  to  determine  trends  within  and  between  program 
categories  -  and  longitudinally  across  the  past  20  or  more  years.  The  objective  of  the 
Phase  II  study  is  to  determine  /  demonstrate  the  value  added  from  early  and  effective 
system  training  practices  and  the  life  cycle  benefits  to  detailed  and  rigorous  insertion  of 
system  training  details  early  on,  and  continuously  in  all  phases  the  acquisition  process. 
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Training  Systems  Study  -  Discussion 


Phase  I 

Initial  Study  Phase 

-  Determine  variables  of  study 

-  Identify  sources  of  technical 
documents 

Execution  Phase 

-  Assemble  and  code  documents 

-  Populate  database 

Analysis  Phase  (in  process) 

-  Analyze  database  for  relationships 
&  trends 

-  Identify  gaps  and  shortcomings 


Phase  II 

*  Extend  Phase  I  database  for  historic 
examples 

*  Determine  specific  metrics  for  assessing 
early  and  effective  system  training  practices 
in  acquisition 

*  Select  systems  for  case  study 

-  Assess  the  impact  of  early  and 
effective  system  training  practices 

-  Determine  trends  within  and 
between  program  categories 

Demonstrate  value  added  from  early  and 
effective  system  training  practices  and  life 
cycle  cost  benefits 


5/26/2011 
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APPENDIX  A 


POLICY  AND  GUIDANCE  EXCERPTS 

The  following  are  excerpts  of  policy  and  guidance  for  development  of  training 
capabilities  associated  with  acquisition  programs.  It  was  noted  that  the  requirement  to 
develop  training  systems  plans  (TSP)  in  DoDI  5000.02  does  not  specify  by  what  point  in 
the  acquisition  process  these  plans  should  be  developed.  Requiring  submission  of  the 
TSPs  early  in  program  development  would  facilitate  improved  effectiveness  of  the 
training  systems  provided. 

DoDI  5000.02  (Operation  of  the  Defense  Acquisition  System) 

8.  OPERATIONS  AND  SUPPORT  PHASE 

a.  Purpose.  The  purpose  of  the  Operations  and  Support  Phase  is  to  execute  a 
support  program  that  meets  materiel  readiness  and  operational  support  performance 
requirements,  and  sustains  the  system  in  the  most  cost-effective  manner  over  its  total  life 
cycle.  Planning  for  this  phase  shall  begin  prior  to  program  initiation  and  shall  be 
documented  in  the  ECSP.  Operations  and  Support  has  two  major  efforts,  Eife-Cycle 
Sustainment  and  Disposal.  The  DoDI  requires  that  Program  Managers  do  the  following  in 
support  for  sustainment  of  the  system: 

(c)  Effective  sustainment  of  systems  results  from  the  design  and  development  of 
reliable  and  maintainable  systems  through  the  continuous  application  of  a  robust  systems 
engineering  methodology. 

2.  Optimize  operational  readiness  via: 

a.  Human-factors  engineering  to  design  systems  that  require  minimal  manpower; 
provide  effective  training;  can  be  operated  and  maintained  by  users;  and  are  suitable 
(habitable  and  safe  with  minimal  environmental  and  occupational  health  hazards)  and 
survivable  (for  both  the  crew  and  equipment). 

b.  Diagnostics,  prognostics,  and  health  management  techniques  in  embedded  and 
off-equipment  applications  when  feasible  and  cost-effective  (Reference  (o)); 

c.  Embedded  training  and  testing,  with  a  preference  for  approved  DoD  Automatic 
Test  Systems  (ATS)  Eamilies  to  satisfy  ATS  requirements; 

d.  Serialized  item  management  techniques  and  the  use  of  automatic  identification 
technology  (AIT),  radio-frequency  identification,  and  iterative  technology  refreshment. 
PMs  shall  ensure  that  data  syntax  and  semantics  for  high-capacity  AIT  devices  conform 
to  International  Organization  for  Standardization  ISO  15418  and  ISO  15434  (References 
(t)  and  (u)). 


from  ENCEOSURE  7:  RESOURCE  ESTIMATION 
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5.  ANALYSIS  OF  ALTERNATIVES  PROCEDURES.  Eor  potential  and 
designated  ACAT  I  and  lA  programs,  the  DPA&E  shall  draft,  for  MDA  approval,  AoA 
study  guidance  for  review  at  the  Materiel  Development  Decision,  hollowing  approval, 
the  guidance  shall  be  issued  to  the  DoD  Component  designated  by  the  MDA,  or  for 
ACAT  lA  programs,  to  the  office  of  the  Principal  Staff  Assistant  responsible  for  the 
mission  area.  The  DoD  Component  or  the  Principal  Staff  Assistant  shall  designate 
responsibility  for  completion  of  the  study  plan  and  the  AoA;  neither  of  which  may  be 
assigned  to  the  PM.  The  study  plan  shall  be  coordinated  with  the  MDA  and  approved  by 
the  DPA&E  prior  to  the  start  of  the  AoA.  The  final  AoA  shall  be  provided  to  the  DPA&E 
not  later  than  60  days  prior  to  the  DAB  or  ITAB  milestone  reviews.  The  DPA&E  shall 
evaluate  the  AoA  and  provide  an  assessment  to  the  Head  of  the  DoD  Component  or 
Principal  Staff  Assistant  and  to  the  MDA.  In  this  evaluation,  the  DPA&E,  in 
collaboration  with  the  OSD  and  Joint  Staff,  shall  assess  the  extent  to  which  the  AoA: 

a.  Illuminated  capability  advantages  and  disadvantages; 

b.  Considered  joint  operational  plans; 

c.  Examined  sufficient  feasible  alternatives; 

d.  Discussed  key  assumptions  and  variables  and  sensitivity  to  changes  in  these; 

e.  Calculated  costs;  and, 

f.  Assessed  the  following: 

(1)  Technology  risk  and  maturity; 

(2)  Alternative  ways  to  improve  the  energy  efficiency  of  DoD  tactical  systems 
with  end  items  that  create  a  demand  for  energy,  consistent  with  mission  requirements  and 
cost  effectiveness;  and 

(3)  Appropriate  system  training  to  ensure  that  effective  and  efficient  training  is 
provided  with  the  system. 


from  ENCEOSURE  8:  HUMAN  SYSTEMS  INTEGRATION  (HSI) 

1.  GENERAE.  The  PM  shall  have  a  plan  for  HSI  in  place  early  in  the  acquisition 
process  to  optimize  total  system  performance,  minimize  total  ownership  costs,  and  ensure 
that  the  system  is  built  to  accommodate  the  characteristics  of  the  user  population  that  will 
operate,  maintain,  and  support  the  system. 

2.  HSI  PEANNING.  HSI  planning  shall  be  summarized  in  the  Acquisition 
Strategy  and  SEP  and  shall  address  the  following: 

e.  Training.  The  PM  shall  work  with  the  training  community  to  develop  options 
for  individual,  collective,  and  joint  training  for  operators,  maintainers  and  support 
personnel,  and,  where  appropriate,  base  training  decisions  on  training  effectiveness 
evaluations.  The  PM  shall  address  the  major  elements  of  training,  and  place  special 
emphasis  on  options  that  enhance  user  capabilities,  maintain  skill  proficiencies,  and 
reduce  individual  and  collective  training  costs.  The  PM  shall  develop  training  system 
plans  to  maximize  the  use  of  new  learning  techniques,  simulation  technology,  embedded 
training  and  distributed  learning  (DoD  Instruction  1322.26  (Reference  (be))),  and 
instrumentation  systems  that  provide  “anytime,  anyplace”  training  and  reduce  the 
demand  on  the  training  establishment.  Where  possible,  the  PM  shall  maximize  the  use  of 
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simulation- supported  embedded  training,  and  the  training  systems  shall  fully  support  and 
mirror  the  interoperability  of  the  operational  system  (DoD  Directive  1322.18  (Reference 
(bf))). 


DoDD  1322.18:  Military  Training 

4.  POLICY.  It  is  DoD  policy  that: 

a.  Individual,  staff,  and  collective  military  training  programs  funded  by  the 
Department  of  Defense  shall  be  available  to  Active  and  Reserve  Component  personnel, 
civilian  employees  and,  when  authorized,  to  contractors,  allies,  and  other  U.S. 
Government  or  non- Government  agency  personnel. 

b.  Military  training  to  generate  and  sustain  capabilities  required  by  the  CCDRs 
and  Chairman  of  the  Joint  Chief  of  Staff  Concept  (Reference  (b))  shall  encompass  all 
phases  of  joint  campaigns  and  the  full  range  of  integrated  operations. 

c.  Members  of  the  Department  of  Defense  shall  receive,  to  the  maximum  extent 
possible,  timely  and  effective  individual,  collective,  and  staff  training,  conducted  in  a  safe 
manner,  to  enable  performance  to  standard  during  operations. 

d.  The  Department  of  Defense  shall  maintain  a  comprehensive  and  effective 
Service,  Defense  Agency,  and  joint  training  management  capability  to  develop,  execute, 
and  assess  military  training  throughout  the  Department. 

e.  Training  capabilities  shall  be  based  on  a  DoD  training  architecture  and  an  open, 
net-centric,  interoperable  standard. 

f.  Embedded  training  and  development  of  net-centric  training  capabilities  shall  be 
considered  as  the  first  alternative  for  cost-effective  delivery  of  instruction  and  training. 


DoDI  1322.26:  Development,  Management,  and  Delivery  of  Distributed 
Learning 

4.  POLICY 

It  is  DoD  policy  that: 

4.1.  Embedded  training  and  distributed  learning  shall  be  considered  as  the  first 
option  to  meet  training  requirements  of  defense  technology  projects  and  acquisition 
programs  (Reference  (a)). 

Acquisition  Guide  Book 

3.4.2. 1.  Cost  Analysis  Requirements  Description  (CARD) 

System  support  concept 

•  System  logistics  concept 

•  Hardware  maintenance  and  support  concept 

•  Software  support  concept 

•  System  training  concept 
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It  also  is  important  that  the  analytic  approach  to  the  cost  estimate  be  documented 
and  reviewed  by  all  potentially  interested  parties,  before  the  actual  work  on  preparing  the 
cost  estimate  begins.  This  helps  ensure  that  there  are  no  false  starts  or  misunderstandings 
later  in  the  process.  Normally,  cost  estimates  are  sponsored  by  a  system  program  office 
and  are  prepared  by  a  multi-disciplinary  team  with  functional  skills  in  financial 
management,  logistics,  engineering,  and  other  talents.  The  team  also  should  include 
participants  or  reviewers  from  major  affected  organizations,  such  as  the  system’s 
operating  command,  product  support  center,  maintenance  depot,  training  center  or 
command,  and  so  forth.  Typically,  the  analytic  approach  to  the  cost  estimate  has  a  written 
study  plan  that  includes  a  master  schedule  (of  specific  tasks,  responsible  parties,  and  due 
dates).  For  sufficiently  complex  efforts,  the  estimating  team  may  be  organized  as  a 
formal  Integrated  Product  Team  (IPT).  For  independent  cost  estimates,  the  team  may  be 
smaller  and  less  formal,  but  the  basic  principle — complete  coordination  of  the  analytic 
approach  with  all  interested  parties — still  applies. 

4.3. 1.3.1.  Interpret  User  Needs;  Analyze  Operational  Capabilities  and 
Environmental  Constraints 

This  step  includes  the  aggregation  of  all  inputs  available  at  this  stage  of  the 
program  (Initial  Capabilities  Document,  Analysis  of  Alternatives  Plan,  exit  criteria  for 
the  phase,  concept  alternatives  for  overall  tactical  system,  as  well  as  associated  support 
system,  training  system,  and  interoperable  systems). 

4.3. 1.3.3.  Decompose  Concept  Performance  into  Functional  Definition  and 
Verification  Objectives 

This  step  includes  the  further  decomposition  of  concept  system  performance  to 
the  functional  level.  Consideration  should  be  given  to  inclusion  of  functionality  and 
functional  flow  definition  across  the  full  system  concept  (tactical  system,  support  system, 
training  system)  and  how  this  functionality  relates  to  other  interoperable  systems 
(functional  interfaces).  Critical  to  this  analysis  is  an  understanding  of  the  level  of 
functionality  achievable  within  program  constraints  and  risk.  Trade  space  and  risk  should 
be  analyzed  and  assessed  against  desired  functional  performance.  Trade  offs  are  made  to 
stay  within  program  constraints  and  may  require  changes  to  higher-level  system  or 
concept  definitions. 

4.3.2.3.1.  Interpret  User  Needs;  Analyze  Operational  Capabilities  and 
Environmental  Constraints 

This  step  includes  the  aggregation  of  all  inputs  available  at  this  stage  of  the 
program  (Initial  Capabilities  Document,  draft  Capability  Development  Document,  results 
of  the  Analysis  of  Alternatives  and  identification  of  the  preferred  system  concept,  exit 
criteria  for  the  phase.  Systems  Engineering  Plan,  Technology  Development  Strategy,  Test 
and  Evaluation  Strategy,  as  well  as  associated  support  and  maintenance  concepts  and 
technologies,  training  system,  and  interoperable  systems). 

4. 3. 2. 3. 3.  Develop  Eunctional  Definitions  for  Enabling/Critical  Technologies  and 
Associated  Verification  Plan 
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This  step  requires  the  further  decomposition  of  system  performance  to  the 
functional  level.  The  functional  requirements  should  be  evaluated  against  available 
technologies,  such  that  enabling  and/or  critical  technologies  can  be  defined. 
Consideration  should  be  given  to  inclusion  of  functionality  and  functional  flow  definition 
across  the  full  system  (tactical  system,  support  system,  training  system)  and  how  this 
functionality  relates  to  other  interoperable  systems  (functional  interfaces).  Critical  to  this 
analysis  is  an  understanding  of  the  level  of  functionality  achievable  within  the  program 
constraints  and  program  risk.  Trade  space  and  risk  should  be  analyzed  and  assessed 
against  desired  functional  performance.  Trade-offs  may  be  required  to  stay  within 
program  constraints  and  may  require  changes  to  higher-level  system  definitions. 

4. 3. 3.4. 4.  Preliminary  Design  Review  (PDR) 

The  PDR  is  a  multi-disciplined  technical  review  to  ensure  that  the  system  under 
review  can  proceed  into  detailed  design,  and  can  meet  the  stated  performance 
requirements  within  cost  (program  budget),  schedule  (program  schedule),  risk,  and  other 
system  constraints.  Generally,  this  review  assesses  the  system  preliminary  design  as 
captured  in  performance  specifications  for  each  configuration  item  in  the  system 
(allocated  baseline),  and  ensures  that  each  function  in  the  functional  baseline  has  been 
allocated  to  one  or  more  system  configuration  items.  Configuration  items  may  consist  of 
hardware  and  software  elements  and  include  such  items  as  airframes,  avionics,  weapons, 
crew  systems,  engines,  trainers/training,  etc. 

6.2.3  Training 

6.2.3. 1  Training  Overview 

Training  is  the  learning  process  by  which  personnel  individually  or  collectively 
acquire  or  enhance  predetermined  job-relevant  knowledge,  skills,  and  abilities  by 
developing  their  cognitive,  physical,  sensory,  and  team  dynamic  abilities.  The 
"training/instructional  system"  integrates  training  concepts  and  strategies  and  elements  of 
logistic  support  to  satisfy  personnel  performance  levels  required  to  operate,  maintain,  and 
support  the  systems.  It  includes  the  "tools"  used  to  provide  learning  experiences  such  as 
computer-based  interactive  courseware,  simulators,  and  actual  equipment  (including 
embedded  training  capabilities  on  actual  equipment),  job  performance  aids,  and 
Interactive  Electronic  Technical  Manuals. 

6. 2. 3.2  Training  Parameters/Requirements 

When  developing  the  training/instructional  system,  the  program  manager  should 
employ  transformational  training  concepts,  strategies,  and  tools  such  as  computer  based 
and  interactive  courseware,  simulators,  and  embedded  training  consistent  with  the 
strategy,  goals  and  objectives  of  the  Training  Transformation  Strategic  Plan  (March  1, 
2002)  and  the  Training  Transformation  Implementation  Plan  and  Appendix  1  (June 
2004).  The  Departments  vision  for  raining  Transformation  is  to  provide  dynamic, 
capabilities-based  training  in  support  of  national  security  requirements  across  the  full 
spectrum  of  Service,  joint,  interagency,  intergovernmental,  and  multinational  operations. 
This  new  approach  emphasizes  the  mission  requirements  of  the  combatant  commanders 
(COCOM).  The  COCOM  is  the  customer.  The  intent  is  to  design  systems  and  structure 
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acquisition  programs  focused  on  the  training  needs  of  the  COCOM.  The  desired  outcome 
is  to  fully  support  COCOM  requirements,  missions,  and  capabilities,  while  preserving  the 
ability  of  the  DoD  Components  to  train  for  their  core  competencies.  The  Under  Secretary 
of  Defense  for  Personnel  and  Readiness,  as  a  member  of  the  Defense  Acquisition  Board, 
assesses  the  ability  of  the  acquisition  program  to  support  the  Military  Departments, 
COCOMs,  and  DoD  Components.  "Training,"  in  this  context,  includes  training, 
education,  and  job-performance  aiding.  Joint  training  must  be  able  to  support  a  broad 
range  of  roles  and  responsibilities  in  military,  multinational,  interagency,  and 
intergovernmental  contexts,  and  the  Department  of  Defense  must  provide  such  training  to 
be  truly  flexible  and  operationally  effective.  Training  readiness  will  be  assessed  and 
reported,  not  only  in  the  traditional  joint  context,  but  also  in  view  of  this  broader  range  of 
"joint"  operations.  Joint  training  and  education  will  be  recast  as  components  of  lifelong 
learning  and  made  available  to  the  Total  Force-active,  reserve,  and  DoD  civilians.  The 
Department  will  expand  efforts  to  develop  officers  well  versed  in  joint  operational  art. 
The  interfaces  between  training  systems  and  the  acquisition  process  will  be  strengthened. 
The  Under  Secretary  of  Defense  for  Personnel  and  Readiness,  as  a  member  of  the 
Defense  Acquisition  Board,  assesses  an  acquisition  program's  ability  to  support  the 
Combatant  Commander's  and  DoD  Components'  capabilities  to  provide  HSI  as  an 
integral  part  of  an  acquisition  program. 

The  program  manager  should  summarize  major  elements  of  the  training  plan  in 
the  Support  Strategy.  This  should  include  logistics  support  planning  for  training,  training 
equipment  and  training  device  acquisitions  and  installations. 

A  Special  Note  on  Embedded  Training.  Both  the  sponsor  and  the  program 
manager  should  give  careful  consideration  and  priority  to  the  use  of  embedded  training  as 
defined  in  DoD  Directive  1322.18:  “Capabilities  built  into,  strapped  onto,  or  plugged  into 
operational  materiel  systems  to  train,  sustain,  and  enhance  individual  and  crew  .skill 
proficiencies  necessary  to  operate  and  maintain  the  equipment.”  The  sponsor’s  decisions 
to  use  embedded  training  should  be  made  very  early  in  the  capabilities  determination 
process.  Analysis  should  be  conducted  to  compare  the  embedded  training  with  more 
traditional  training  media  (e.g.,  simulator  based  training,  traditional  classroom 
instruction,  and/or  maneuver  training)  for  consideration  of  a  system’s  Total  Operating 
Cost.  The  analysis  should  compare  the  costs  and  the  impact  of  embedded  training  (e.g., 
training  operators  and  maintenance  personnel  on  site  compared  to  off  station  travel  to  a 
temporary  duty  location  for  training).  It  should  also  compare  the  learning  time  and  level 
of  effectiveness  (e.g.,  higher  “kill”  rates  and  improved  maintenance  times)  achieved  by 
embedded  training.  When  making  decisions  about  whether  to  rely  exclusively  on 
embedded  training,  analysis  must  be  conducted  to  determine  the  timely  availability  of 
new  equipment  to  all  categories  of  trainees  (e.g..  Reserve  and  Active  Component  units  or 
individual  members).  For  instance,  a  National  Guard  tank  battalion  that  stores  and 
maintains  its  tanks  at  a  central  maintenance/training  facility  may  find  it  more  cost 
effective  to  rely  on  mobile  simulator  assets  to  train  combat  tasks  rather  than  transporting 
its  troops  to  the  training  facility  during  drill  weekends.  A  job  aid  for  embedded  training 
costing  and  effectiveness  analyses  is:  “A  Guide  for  Early  Embedded  Training  Decisions,” 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  Research  Product 
96-06. 
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6. 2. 3. 3  Training  Planning 

This  section  will  prepare  the  Program  Manager  to  understand  training  capabilities 
as  an  integral  part  of  the  Joint  Capabilities  Integration  and  Development  System  and, 
with  assistance  of  the  training  community,  translate  those  capabilities  into  system  design 
features. 

First,  the  Joint  Capabilities  Integration  and  Development  System  process  should 
address  joint  training  parameters  for  military  (Active,  Reserve,  and  Guard)  and  civilian 
personnel  who  will  operate,  maintain,  and  support  the  system.  Training  programs  should 
employ  a  cost-effective  solution,  consisting  of  a  blend  of  capabilities  that  use  existing 
training  programs  and  introduces  new  performance-based  training  innovations.  This  may 
include  requirements  for  school  and  unit  training,  as  well  as  new  equipment  training,  or 
sustainment  training.  This  also  may  include  requirements  for  instructor  and  key  personnel 
training  and  new  equipment  training  teams. 

Training  should  be  considered  early  in  the  capabilities  development  process 
beginning  with  the  analyses  that  supports  development  of  the  Initial  Capabilities 
Document  and  continues  with  development  of  the  Capability  Development  Document. 

The  Capability  Development  Document  should  discuss  the  specific  system 
training  requirements: 

•  Allow  for  interactions  between  platforms  or  units  (e.g.,  through  advanced 
simulation  and  virtual  exercises)  and  provide  training  realism  to  include  threats 
(e.g.,  virtual  and  surrogate),  a  realistic  electronic  warfare  environment, 
communications,  and  weapons. 

•  Embedded  training  capabilities  that  do  not  degrade  system  performance  below 
threshold  values  nor  degrade  the  maintainability  or  component  life  of  the  system. 

•  That  Initial  Operational  Capability  is  attained  and  that  training  capabilities  are 
embedded  and  met  by  Initial  Operational  Capability. 

•  An  embedded  performance  measurement  capability  to  support  immediate 
feedback  to  the  operators/maintainers  and  possibly  to  serve  as  a  readiness 
measure  for  the  unit  commander. 

•  Training  logistics  necessary  to  support  the  training  concept(e.g.,  requirements  for 
new  or  upgrades  to  existing  training  facilities). 

•  The  training  community  should  be  specific  in  translating  capabilities  into  system 
requirements.  They  should  also  set  training  resource  constraints.  Examples  are: 

•  The  training  community  should  consider  whether  the  system  be  designed  with  a 
mode  of  operation  that  allows  operators  to  train  interactively  on  a  continuous 
basis,  even  when  deployed  in  remote  locations. 

•  The  training  community  should  consider  whether  the  system  be  capable  of 
exhibiting  fault  conditions  for  a  specified  set  of  failures  to  allow  rehearsal  of 
repair  procedures  for  isolating  faults  or  require  that  the  system  be  capable  of 
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interconnecting  with  other  (specific)  embedded  trainers  in  both  static  and 
employed  conditions. 

•  The  training  community  should  consider  whether  embedded  training  capabilities 
allow  enhancements  to  live  maneuver  such  that  a  realistic  spectrum  of  threats  is 
encountered  (e.g.,  synthetic  radar  warnings  generated  during  flight). 

•  The  training  community  should  consider  whether  the  integrated  training  system 
be  fully  tested,  validated,  verified,  and  ready  for  training  at  the  training  base  as 
criteria  for  declaring  Initial  Operational  Capability. 

•  From  the  earliest  stages  of  development  and  as  the  system  matures,  the  program 
manager  should  emphasize  training  requirements  that  enhance  the  user’s 
capabilities,  improve  readiness,  and  reduce  individual  and  collective  training  costs 
over  the  life  of  the  system.  This  may  include  requirements  for  expert  systems, 
intelligent  tutors,  embedded  diagnostics,  virtual  environments,  and  embedded 
training  capabilities.  Examples  of  training  that  enhances  user’s  capabilities 
follow: 

•  Interactive  electronic  technical  manuals  provide  a  training  forum  that  can 
significantly  reduce  schoolhouse  training  and  may  require  lower  skill  levels  for 
maintenance  personnel  while  actually  improving  their  capability  to  maintain  an 
operational  system; 

•  Requirements  for  an  embedded  just-in-time  mission  rehearsal  capability 
supported  by  the  latest  intelligence  information  and  an  integrated  global  training 
system/network  that  allows  team  training  and  participation  in  large  scale  mission 
rehearsal  exercises  can  be  used  to  improve  readiness. 

•  In  all  cases,  the  paramount  goal  of  the  training/instructional  system  should  be  to 
develop  and  sustain  a  ready,  well-trained  individual/unit,  while  giving  strong 
consideration  to  options  that  can  reduce  Life-cycle  costs  and  provide  positive 
contributions  to  the  joint  context  of  a  system,  where  appropriate. 

•  Training  devices  and  simulators  are  systems  that,  in  some  cases,  may  qualify  for 
their  own  set  of  HSI  requirements.  For  instance,  the  training  community  may 
require  the  following  attributes  of  a  training  simulator: 

•  Accommodate  “the  central  90  percent  of  the  male  and  female  population  on 
critical  body  dimensions;” 

•  Not  increase  manpower  requirements  and  should  consider  reductions  in 
manpower  requirements; 

•  Consider  reduced  skill  sets  to  maintain  because  of  embedded  instrumentation; 

•  Be  High  Level  Architecture  compliant; 

•  Be  Sharable  Content  Object  Reference  Model  compliant; 

•  Be  Test  and  Training  Enabling  Architecture  compliant; 
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•  Use  reusable  simulation  objects. 


CJCSI  3170.01G:  JCIDS 

2.  Introduction  to  the  JCIDS  Process 

e.  During  the  Technology  Development  phase  (Figure  A-4),  the  sponsor  performs 
technology  maturation  activities,  builds  competitive  prototypes,  and  may  perform  design 
activities  leading  to  a  preliminary  design  review.  The  ICD  provides  a  wide  aperture  for 
operational  capability  to  define  system  requirements  and  to  encourage  technological 
innovation.  It  is  vital  the  science  and  technology,  users,  training,  and  system  developer 
communities  collaborate  to  agree  on  a  proposed  solution  that  is  affordable,  militarily 
useful,  and  based  on  mature,  demonstrated  technology. 

JCIDS  Manual 

d.  Selectively  Applied  KPPs.  The  JROC  has  defined  two  KPPs  to  be  selectively 
applied  to  programs:  system  training  and  energy  efficiency.  The  sponsor  will  perform  an 
analysis  on  the  use  of  these  parameters  as  KPPs.  If  the  analysis  determines  that  they 
should  not  be  KPPs,  a  summary  of  the  justification  will  be  provided  in  the  CDD. 

(1)  System  Training  KPP.  Training  should  be  considered  early  in  the  capabilities 
development  process  beginning  with  the  analyses  that  support  development  of  the  ICD 
and  continues  with  development  of  the  CDD.  Ensure  system  training  is  addressed  in  the 
AoA  and  supporting  analysis  for  subsequent  acquisition  phases  and  ensure  projected 
training  requirements  and  associated  costs  are  appropriately  addressed  across  the  program 
life  cycle.  Embedded  training  and  net-centric  enabled  training  shall  be  considered  the 
first  alternative  for  cost  effective  delivery  of  instruction.  The  training  capability 
requirements  should  be  on  par  with  operational  systems  capability.  Eurther  guidance  on 
this  KPP  can  be  found  in  Appendix  C  to  this  Enclosure. 
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APPENDIX  B 


Variable 

Number 

1 

coding 

2 

coding 

3 

coding 

4 

coding 

5 

coding 

6 

coding 

7 

coding 

8 

coding 

9 

coding 

10 

coding 

11 

coding 

12 

coding 

13 


CODING  RULES 


Database  Entry  # 

Rater  Initials: 

Report  Document  Title 

Full  title,  volume  number  if  appropriate  (enter  in  cell  below) 

ADA  number  (DTIC) 

Enter  number  in  cell  below,  If  no  ADA  number,  enter  0  (in  cell  below) 

Other  reference  (journal,  book  chapter,  conference  paper,  other  report) 

If  DTIC  enter  0;  APA  style,  or  close  to  it  (enter  in  cell  below) 

Pages 

Enter  total  page  count  as  listed  in  document  (enter  in  cell  below) 

Corporate  Author 

Enter  all  authors:  last  name,  first  initial.  If  none  identified,  enter  organization  (enter  in  cell  below) 

Year  of  Report 

The  year  the  report  was  published,  YYYY  (enter  in  cell  below),  enter  0  if  not  reported 
Type  of  Activity  publishing/submitting  report  (not  necessarily  sponsor) 

A=Army,  AF=AirForce,  N=Navy,  M=MarineCr,  D=Def  Agency,  J=Joint,  l=industry,  F=FFRDC,  U=acad 

Funding  for  Report  Document  (possible  R&D  funding  for  report  on  Major  System) 

Usually  a  program  element  is  supplied  in  DTIC  abstract.  0  =  unknown  or  n/a,  R=R&D,  0=C 
P=Procurement,  l=lndustry  internal,  X=other 

System  Name  or  Program  Number  (PNO)  for  more  recent  acquisitions 

Formal  name  or  3  digit  PNO;  If  not  system  specific  enter  1 ,  if  unknown  enter  0  (enter  in  cell  below) 
System  Type  -  Functional  Capability 

0=unknown  or  n/a,  1=Battlespace  awareness,  2=Building  Partnerships,  3=  C2,  4=Corporate  Mngt/Sup 
5=  Force  Application,  6=  Force  Support,  7=Logistics,  8=NetCentric,  9=Protection(examples  available) 

Acquisition  Phase  (as  of  Year  variable) 

0=unknown  or  n/a,  1=Concept  Refinement,  2=TechDevel,  3=SystemDevel,  4=Production,  5=Ops/supf 
Acquisition  Cost  -  Major  Defense  Acquisition  Program 

From  external  source;  in  FY2000$,  $365M  RDTE,  $2.1 9B  procurement.  1=MDAP,  2=not  M 
0=unknown 

Requirement  Source 
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coding  warfare  experimentation,  R&D,  ACTD,  modernization,  upgrade,  P3I,  Tech  Readiness  Levei;  Scale  TB 

1 4  Who  deveioped  training  -  prime,  sub 

coding  0=unknown  or  n/a,  1=prime,  2=subcontract,  3=other 

15  Training  Support  Pian  -  compieteness,  depth,  etc. 

..  Quaiitative  judgment  on  training  pian  to  maximize  the  use  of  new  iearning  techniques,  simui 

technoiogy,  embedded  training,  and  instrumentation  that  reduce  demand  on  the  training  estabiishr 
instructor  training;  HIS  integration  for  training.  Scale  TBD 

16  System  Training  Pian  -  compieteness,  depth,  etc. 

coding  Concepts  specified  for  simuiators,  ciassroom,  distance  iearning.  Scale  TBD 

17  Embedded  Training 

0=unknown  or  n/a,  1=not  inciuded  (perhaps  not  considered),  2=  expiicitiy  considered  bu 
inciuded,  3=inciuded 

1 8  System  Safety/Heaith  Hazards/Survivabiiity  -  training  reiationship  specified 
coding  7  Human  System  Integration  domains  (per  Mike  McCauiey).  Scaie  TBD 

19  Domain  Integratation  -  training  tradeoffs  between  domains 

coding  0=unknown  or  n/a,  1=no  expiicit  consideration,  2=  yes,  expiicit  consideration 

20  Rapid  Acquisition  -  training  consideration 

coding  0=  unknown  or  n/a,  1  =  no  RA,  2=RA,  3=RA  w/expiicit  training  consideration 

21  Spirai  Deveiopment  (or  incrementai  anaiysis,  evoiutionary  acq)  -  training  consideration 
coding  0=  unknown  or  n/a,  1=no  SD,  2  =  SD,  3=SD  w/  expiicit  training  consideration 

22  System  of  Systems  approach  -  training  consideration 

coding  0=  unknown  or  n/a,  1  =not  SoS,  2  =  SoS,  3=SoS  w/  expiicit  training  consideration 

23  Training  Effectiveness  Anaiysis  or  TE  Evaiuation 

coding  0=unknow  or  n/a,  1=none  reported,  2=yes,  reported 

24  Individuai  Training  MOE/MOPs 

coding  Measures  of  effectiveness/performance,  0=unknown  or  n/a,  1=no,  2=yes 

25  Coiiective  MOE/MOPs 

coding  Measures  of  effectiveness/performance,  0=unknown  or  n/a,  1=no,  2=yes 

26  Maintenance  MOE/MOPs  (crew  reiated  &  system  reiated) 

coding  Measures  of  effectiveness/performance,  0=unknown  or  n/a,  1=no,  2=yes 

27  Operationai  Test  Sites 

coding  0=nothing  to  test;  1  =no  testing  done;  2=DevTest;  3=OpTest;  4=DT+OT 

28  Based  on  empiricai  evidence 

coding  0=unknow  or  n/a,  1  =none  reported,  2=yes  (human  performance  data  reported) 

41 


29  Use  of  military  test  subjects 

coding  0=unknown  or  n/a,  1  =no  (non  military  test  subjects  used),  2=yes 

30  Quantifiable  Level  of  Learning  (Kirkpatrick  Model) 

0=unknown,  1=trainee  reaction,  2=learning  outcomes,  3=transfer  to  job  performance,  4=organiza1 
coding  benefit 

31  Based  on  multiple  studies 

coding  0=unknown  or  n/a,  1=only  one  study/experiment,  2=more  than  one  study/experiment 

32  Proprietary  solution  for  training  delivery  (relates  to  S9) 

coding  0=unknown,  1=proprietary,  2=non  proprietary  (open  source.  Govt  owned) 

33  Generalizability  within/across  system  types 

coding  results  apply  equally  to  other  systems  (one  or  more),  such  as  Force  Application  findings  app 

to  the  Logistics  area  (as  identified  in  S2).  0=unknown  or  n/a,  1=yes,  2=no 

34  GAO  Report 

coding  0=  not  GAO,  else  GAO  report  number 

35  Year  of  GAO  Report 

coding  0=not  GAO,  else  report  year 

36  Shortcomings  on  training  (effectiveness,  cost,  currency  etc.) 

coding  in  abstract  form 

END  NOTES  AND  COMMENTS  -  enter  any  observations,  viewpoints  etc.  Open  ended  response, 
coding  What  is  the  impact  on  the  gap  analysis? 
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APPENDIX  C 


TECHNICAL  ABSTRACTS 
Chronological  Order 


Entry  #16 

Training  Effectiveness  Evaiuation  of  Navai  Training  Devices.  Part  1.  A  Study  of 
Submarine  Diving  Trainer  Effectiveness 
1970 
Navy 

SSN594,  SSN613,  SSN627 
Human  performance  data  reported 

Not  generaiizabie 

The  study  had  a  duai  orientation:  that  of  considering  probiems  inherent  in  evaiuating  trainer 
effectiveness  within  the  context  of  the  training  environment,  and  evaiuating  the  specific 
submarine  diving  trainer  for  its  training  effectiveness.  A  comparison  was  made  between 
experienced  student  crew  performance  and  inexperienced  student  crew  performance.  Aii 
training  devices  (SSN613  and  SSN627)  showed  simiiar  improvement  in  performance  after 
severai  iterations,  even  though  the  dispiays  and  controis  of  each  training  device  were  different. 
The  exception  was  the  third  device,  SSN594,  which  was  deemed  ineffective  as  a  substitute 
trainer  for  either  the  SSN613  or  SSN627  crews.  The  resuits  indicated  that  differences  between 
the  dispiays  and  controis  of  the  SSN613  and  SSN627  training  devices  did  not  impact  training 
effectiveness.  Rather  the  difference  was  with  the  effectiveness  of  the  device  itseif.  In  that,  the 
SSN594  device  does  not  have  redundant  indicators  of  criticai  variabies,  such  as  trim  angie,  and 
provided  indications  in  a  compieteiy  different  manner.  Study  data  was  not  sufficient  to  evaiuate 
trainer  effectiveness  of  each  device  for  the  individuai  crews,  as  experienced  crews  were  not 
avaiiabie  for  comparison  to  the  student  crews.  Another  iimitation  of  the  training  effectiveness 
evaiuation  was  the  amount  of  time  in  the  trainer  to  practice  compiex  situations.  Many  emergency 
situations  were  simpiy  demonstrated,  rather  than  practiced,  due  to  iack  of  time. 


Entry  #12 

Using  a  Ground  Trainer  in  a  Job  Sampie  Approach  to  Predicting  Piiot  Performance. 
1971 

Air  Force 
T-37 

Human  performance  data  reported 


This  study  has  severai  weaknesses,  inciuding  a  subjective  performance  measure.  Stiii,  it  has 
the  interesting  notion  of  using  performance  in  iow  cost  simulators  to  performance  on  high  cost, 
high  demand  simulators,  but  the  correlations  are  moderate,  .38  is  the  highest.  The  study 
reflects  knowledge  of  this  training-testing  technique. 


Entry  #15 

Training  Effectiveness  Evaluation  of  Naval  Training  Devices  Part  II:  A  Study  of  Device  2F66A 
(S-2E  Trainer)  Effectiveness 
1971 
Navy 


43 


2F66A  S-2E  Trainer 

Human  performance  data  reported 

Generalizable 

A  study  was  performed  of  the  effectiveness  of  Device  2F66A  (S-2E  aircraft)  in  training  crew 
members  to  perform  air  antisubmarine  warfare  missions.  Aithough  each  conciusion  must  be 
quaiified  by  consideration  of  the  data  sampie  and  detaiied  resuits  described  in  the  report,  it  was 
possibie  to  arrive  at  the  foiiowing  overaii  conciusions: 

a.  The  trainer  was  found  to  be  effective  in  training  aii  crew  members. 

b.  The  trainer  appeared  to  be  more  effective  for  beginning  students  than  for  operationai  and 
reserve  personnei. 

c.  Factors  which  instructors  used  to  vary  the  difficuity  of  training  sessions  did  not  appear  to  have 
a  major  effect  on  overaii  trainee  performance. 

d.  The  effect  of  team  training  sessions  on  student  performance  was  consistent  oniy  during  the 
eariy  part  of  the  first  team  session. 

e.  Instructors  considered  the  trainer  effective  in  training.  However,  equipment  maifunctions  and 
insufficient  fideiity  have  reduced  confidence  and  iowered  the  overaii  acceptabiiity  of  the  device 
as  an  integrai  part  of  air  ASW  training. 


Entry  #17 

Training  Effectiveness  Evaiuation  of  Navai  Training  Devices:  An  Evaiuation  of  the 
2F69B  ASW  Weapon  System  Trainer 
1972 
Navy 

2F69B  ASW  Weapon  System 
Human  performance  data  reported 

Generaiizabie 

Two  separate  experiments  were  conducted  to  evaiuate  the  effectiveness  of  the  Navy  2F69B 
Weapon  System  Trainer.  The  first  series  of  experiments  were  conducted  in  the  WST.  The 
second  series  of  experiments  were  conducted  in  both  the  WST  and  airborne  environment.  The 
resuits  showed  that  the  most  beneficiai  training  occurred  in  the  WST  trainer.  As  a  resuit  of  these 
studies,  it  was  recommended  that  the  number  of  WST  training  sessions  be  increased,  and  the 
number  of  training  fiights  be  reduced  accordingiy. 


Entry  #2 

Transfer  of  Training  Effectiveness:  A7E  Night  Carrier  Landing  Device  2F103 

1976 

Navy 

A7 

Human  performance  data  reported 
Generaiizabie 

A  transfer  of  training  evaiuation  found  the  simuiator  effective  compared  controis  receiving  no 
device  training.  Five  objective  measures  of  ianding  performance  measured  during  actuai  carrier 
quaiification  fiights  demonstrated  statisticaiiy  significant  differences  in  ianding  performance, 
notabiy  boarding  rates.  Confirmation  of  the  effectiveness  of  an  A7  night  carrier  ianding  simuiator 
(awareness). 


Entry  #10 
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Training  Effectiveness  Evaiuation  of  Device  2F87F,  P-3C  Operationai  Fiight  Trainer 

1977 

Navy 

P-3C 

Human  performance  data  reported 
Generaiizabie 

A  training  effectiveness  evaiuation  of  P-3C  operationai  fiight  trainer  was  tested  for  effectiveness 
through  a  controiied  transfer  of  training  study.  Measurements  were  checkfiight  grades,  number 
of  fiights  and  iandings  to  criterion  proficiency,  number  of  check  tasks,  and  errors  during  ianding. 
Those  piiots  with  six  sessions  of  simuiator  training  reduced  fiight  hours  by  43  percent  compared 
to  controi  groups.  This  documents  the  vaiue  of  simuiation  for  fiight  training  (awareness). 


Entry  #3 

A  Strategy  for  the  Deveiopment  of  Training  Devices 

1978 

Air  Force 

AC-130E 

Human  performance  data  reported 
Generaiizabie 

Describes  a  methodoiogy  to  aid  simuiator  and  training  device  design,  empioying  users,  training 
psychoiogists,  and  simuiation  engineers.  Empiricai  testing  on  four  crew  positions  for  AC-130 
gunship  demonstrated  a  moderate  ievei  of  effectiveness;  vaiue  as  design  tooi  for 
impiementation. 


Entry  #1 1 

Task-specific  Simuiator  Pretraining  and  In-fiight  Stress  of  Student  Piiots 

1978 

Air  Force 

T-37 

Human  performance  data  reported 
Generaiizabie 

Investigation  of  simuiated  spin  recovery  training  in  reducing  stress  during  first  in-fiight  exposure 
to  power-on  staiis  during  piiot  training. 


Entry  #29 

Integration  and  Appiication  of  Human  Resource  Technoiogies  in  Weapon  System  Design: 
Consoiidated  Data  Base  Specification 
1978 
Air  Force 
1 

Human  performance  data  reported 
Generaiizabie 
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Effort  resulted  in  the  detailed  specification  of  the  consolidated  data  base  with  a  description  of 
steps  necessary  for  development  and  maintenance.  The  consolidated  Human  Resource 
Technologies  to  encompass  maintenance  manpower,  instructional  system  development,  job 
guide  development,  system  ownership  costs  and  human  resources  in  design  tradeoffs.  Report 
outlines  major  data  storage  for  all  areas  identified  above. 


Entry  #5 

Boom  Operator  Part-Task  Trainer:  Test  and  Evaluation  of  the  Transfer  of  Training 

1979 

Air  Force 

KG-135 

Human  performance  data  reported 
Not  generalizable 

Reports  the  rationale,  methodology  and  results  of  a  study  of  the  training  capabilities  of  the  Boom 
Operator  Part-Task  Trainer  (BOPPT),  an  air  refueling  simulator.  The  study  was  performed  as 
part  of  an  Air  Force  Initial  Operational  Test  and  Evaluation  (lOT&E)  of  the  device,  it  was  found 
that  all  student  groups  trained  in  the  BOPTT  required  significantly  fewer  air  refueling  attempts 
(50  versus  71)  to  reach  proficiency  in  KC-135A  air  refueling  skills  than  did  conventionally  trained 
students.  This  contributes  to  knowledge  and  awareness  for  transfer  of  training  from  simulators 
to  aircraft. 


Entry  #7 

Training  Effectiveness  of  the  CH-47  Flight  Simulator. 

1979 

Army 

CH-47 

Human  performance  data  reported 
Generalizable 

Test  of  a  prototype  helicopter  flight  simulator.  Results  of  controlled  experiments  indicated 
overall  training  value,  but  difficulty  training  maneuvers  close  to  the  ground  due  to  limitations  of 
the  visual  system.  Good  example  of  value  of  testing  training  with  feedback  for  future  simulator 
design.  Value  for  rapid  acquisition  of  training  simulators. 


Entry  #13 

Transfer  of  Training  Effectiveness  Evaluation:  U.S.  Navy  Device  2B35 

1979 

Navy 

TA-4J 

Human  performance  data  reported 


This  report  describes  a  four-phase  transfer  to  training  effectiveness  evaluation  for  Navy  Device 
2B35,  a  computer-generated  image  visual  display  system  used  in  the  Advanced  Jet  phase  of 
Navy  Undergraduate  Pilot  Training.  The  transfer  results  provide  support  for  familiarization 
maneuvers  but  do  not  support  its  use  for  night  familiarization. 
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Entry  #14 

Final  Report:  Human  Factors  Research  in  Military  Organizatoins  and  Systems 

1979 

Army 


Human  performance  data  reported 
Not  generalizable 

This  report  summarizes  work  on  seven  research  projects:  Target  handoff  techniques;  tank  crew 
measures  of  effectiveness;  fatigue  effects  of  NAV  goggle  use;  long  range  target  recognition; 
effects  of  stress  on  performance;  symbology  for  automated  graphic  displays;  and  suppression 
research.  In  the  study  of  long  range  target  detection  of  armored  vehicle  patterns,  crewmen 
increased  their  recognition  performance  from  47%  in  pretraining  to  79%  during  training  and  90% 
in  the  posttest  phase. 


Entry  #19 

Human  Resources,  Logistics,  &  Cost  Factors  in  Weapon  System  Development: 
Demonstration  in  Conceptual  &  Validation  Phases  of  Aircraft  System  Acquisition 
1979 
Air  Force 

Advanced  Medium  STOL  (short  takeoff  and  landing)  Transport 
Human  performance  data  reported 

Generalizable 

Effort  resulted  in  the  detailed  specification  of  the  consolidated  data  base  with  a  description  of 
steps  necessary  for  development  and  maintenance.  The  consolidated  Human  Resource 
Technologies  to  encompass  maintenance  manpower,  instructional  system  development,  job 
guide  development,  system  ownership  costs  and  human  resources  in  design  tradeoffs.  Report 
outlines  major  data  storage  for  all  areas  identified  above. 


Entry  #4 

Training  Device  Effectiveness:  Formulation  and  Evaluation  Methodology 

1980 

Army 

AH-1 

Human  performance  data  reported 
Generalizable 

Test  of  'cumulative  transfer  effectiveness  ratio"  which  is  the  ratio  of  simulator  training  to  actual 
aircraft  training.  21  rated  helicopter  pilots  were  measured  on  trials  to  criterion  for  performance 
on  31  specific  maneuvers.  A  control  group  had  no  simulator  training.  25  to  80  percent  of 
variance  in  aircraft  training  could  be  accounted  for  by  the  amount  of  prior  simulator  training. 
This  provides  evidence  for  the  training  efficiency  afforded  by  simulators. 


Entry  #30 

Human  Resources,  Logistics,  and  Cost  Factors  in  Weapon  System  Development: 
Methodology  and  Data  Requirements 
1980 
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Air  Force 
0 

Fluman  performance  data  reported 
Generalizable 

This  report  describes  a  methodoiogy  which  is  usefui  for  appiying  human  resources,  iogistics, 
and  cost  factors  in  weapon  system  acquisition  programs.  The  methodoiogy,  termed  the 
coordinated  human  resources  technoiogy  (CFIRT),  was  deveioped  from  an  integration  of  five 
individuai  human  resources  technoiogies:  maintenance  manpower  modeiing,  instructionai 
system  deveiopment,  job  guide  deveiopment,  human  resources  in  design  tradeoffs,  and  system 
ownership  costing.  The  CFIRT  methodoiogy  operates  from  a  consoiidated  data  base  (CDB) 
which  integrates  data  from  these  five  technoiogies  into  a  singie  data  source.  The  CHRT 
methodoiogy  has  two  distinct  capabiiities:  (a)  it  can  assess  the  impact  of  baseiine  and 
aiternative  equipment  designs  and  support  pians  on  human  resources,  iogistics,  and  ownership 
costs,  and  (b)  it  can  provide  an  integrated  maintenance  personnei,  training,  and  technicai 
manuai  program  for  the  system.  The  CHRT  methodoiogy  is  impiemented  by  a  number  of 
computer  operated  modeis,  manuai  operated  modeis,  and  task  anaiysis  procedures.  The  CDB 
has  both  computer  data  fiies  and  hard  copy  data  fiies.  The  description  of  the  CHRT  and  the  CDB 
is  based  upon  a  conceptuai  structure  and  the  resuits  of  a  preiiminary  tryout  of  the  conceptuai 
structure  using  data  from  the  Advanced  Medium  STOL  Transport  System  acquisition  program. 
This  methodoiogy  is  appiicabie  for  tasks  that  are  discrete,  stabie  and  weii  defined. 


Entry  #46 

HUMAN  FACTORS  AND  TRAINING  EVALUATION  OF  THE  IMPROVED  TOW 
VEHICLE  (ITV)  DURING  OT/DT  III 

1980 

Army 

Improved  TOW  vehicle. 

Human  performance  data  reported 

Generalizable 

The  research  was  requested  directly  by  the  TRADOC  System  Manager  for  the  Improved  TOW 
Vehicle  (TSM-ITV)  and  involved  detailed  interaction,  coordination,  and  cooperative  effort  with 
the  Operational  Test  and  Evaluation  Agency  for  Operational  Test  III  and  with  the  Army  Material 
Systems  Analysis  Agency  and  the  Test  and  Evaluation  Command  for  Developmental  Test  III. 
Training  developed  at  the  Armor  and  Infantry  Schools  appeared  adequate  for  the  test 
requirements.  No  negative  transfer  of  training  appeared  between  ground-mounted  TOW  and  ITV 
tracking  systems.  For  knowledge  gap  in  retention  of  cognitive  and  psychomotor  skills:  Skill  loss 
on  ITV  performance  appears  to  be  greater  than  for  TOW  and  gunners  suggested  ITV  tracking 
practice  on  a  weekly  basis.  Test  participants  indicated  that  intervals  between  ITV  practice  for 
skill  maintenance  should  be  no  more  than  one  week.  TOW  practice  sessions  should  be  twice 
monthly.  Individual  gunners  may  require  additional  practice.  Basic  ITV  training  should  require 
five  (5)  days.  An  additional  three  days  is  necessary  for  tactics  training  and  for  operational 
experience. 


Entry  #18 

Human  Resources,  Logistics,  and  Cost  Factors  in  Weapon  System  Development: 
Demonstration  of  Full  Scale  Development  Phase  of  Aircraft  System  Acquisition 
1981 

Air  Force 
0 
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Human  performance  data  reported 
Generalizable 

This  report  documents  the  final  part  of  a  three-part  demonstration  of  the  coordinated  human 
resources  technology  (CHRT)  on  an  aircraft  acquisition  program.  CHRT  is  an  integration  of  five 
human  resource  technologies:  maintenance  manpower  modeling;  human  resource  in  design 
trade-offs,  instructional  system  development,  job  guide  development,  and  system  ownership 
costing.  The  CHRT  methodology  also  includes  a  consolidated  data  base  (CDR)  which  services 
the  five  integrated  technologies.  CHRT  was  conceived  and  developed  (a)  to  assess  the  impact 
of  system  design  and  support  plans  on  human  resource,  logistics,  and  cost  throughout 
acquisition,  and  (b)  to  facilitate  the  implementation  of  an  integrated  personnel,  training  and 
technical  manual  support  approach.  In  this  part  of  the  demonstration,  CHRT  and  the  CDB  were 
applied  to  the  avionics  and  landing  gear  system  of  the  Advanced  Medium  STOL  transport 
(AMST)  using  data  projected  for  the  minimum  engineering  development  phase.  This  phase  may 
be  considered  similar  to  full-scale  development  but  lesser  in  scope.  The  following  results  were 
achieved.  First,  manpower  requirements,  training  requirements,  technical  manual  requirements, 
reliability,  maintainability,  and  system  ownership  costs  were  assessed  and  quantified  for  several 
avionics  and  landing  gear  design  and  support  alternative  at  various  levels  of  system  detail. 
Second,  a  sample  integrated  training  program  and  technical  manual  were  prepared  fro  a 
selected  landing  gear  maintenance  task.  Third  the  consolidated  database  was  continued  and 
expanded  for  both  the  avionics  and  landing  gear  systems. 


Entry  #42 

Preparation  and  design  for  a  training  effectiveness  evaluation  of  device  2F64C  for 
Replacement  Pilot  training 
1981 
Navy 
H-3,  SH-3 

Human  performance  data  reported 


This  report  presents  the  initial  effort  in  a  program  to  assess  the  training  potential  of  a  new 
simulator  (Device  2F64  C)  for  training  SH-3  replacement  helicopter  pilots.  The  report  contains  a 
description  of  the  training  situation  at  the  fleet  readiness  squadron  prior  to  and  during  transition 
to  a  new  curriculum  that  resulted  from  an  instructional  system  development  (ISD)  program.  The 
report  also  discusses  TAEG  initiatives  to  enhance  training  of  replacement  helicopter  pilots  and 
factors  impacting  syllabi  content  and  syllabus  development.  Detailed  scenarios  are  described  to 
ensure  effective  syllabus  implementation.  An  experimental  plan  for  assessing  training 
effectiveness  of  the  2f64C  is  included.  Gap  Analysis:  Classified  as  “best  practice  report”, 
addressing  a  Knowledge  Gap  by  testing  the  efficacy  of  applied  ISD  (task  analysis;  syllabus 
development  and  performance  assessment)  for  a  training  system  prior  to  operational 
acceptance. 

Entry  #57 

Performance  Measurement  and  the  Navy's  Tactical  Aircrew  Training  System  (TACTS) 

1981 

N 

TACTS 

Human  performance  data  reported 

Generalizable 
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The  present  report  was  written  in  response  to  an  air  combat  training  Fieet  Project  Team  (FPT) 
requirement  that  a  performance  measurement  system  (PMS)  for  the  Navy  Tacticai  Aircrew 
Training  System  (TACTS)  be  deveioped.  It  provides  a  context  for  the  probiem  and  vaiue  of 
performance  measurement  as  weii  as  a  set  of  specific  functionai  requirements  for  a  PMS  for  air- 
-to-air  combat  training.  In  addition,  the  report  provides  systematic  documentation  for  the 
conciusion  that,  with  the  deveiopment  of  the  extremeiy  impressive  TACTS  instrumentation 
technoiogy,  there  was  no  paraiiei  deveiopment  to  provide  a  system  of  human  performance 
measurement  to  support  that  capabiiity.  It  is  thought  that  deveiopment  of  a  PMS,  which  meets 
the  requirements  presented  here,  wiii  greatiy  assist  in  more  fuiiy  accompiishing  the  originai 
operationai  training  requirements  which  ied  to  deveiopment  of  the  TACTS  faciiity.  It  is  conciuded 
that  whiie  the  existing  TACTS  represents  a  highiy  advanced  aviation  engineering  technoiogy 
that  can  provide  extremeiy  vaiuabie  training,  that  same  technoiogy  has  iargeiy  ignoring  the 
functionai  requirements  for  a  system  of  human  performance  measurement.  Improvements  in  the 
TACTS  performance  measurement  capabiiities  wiii  improve  its  training  vaiue  even  further. 
Performance  data  are  essentiai  to  the  training  process.  Measures  of  ACM  subtask  skiiis  aiiow 
the  instructor  to  monitor  the  progress  of  training,  as  weii  as  to  provide  diagnostic  feedback 
regarding  probiem  areas.  Both  norm-referenced  and  criterion-referenced  standards  of  aircrew 
proficiency  can  be  estabiished.  These  behaviorai  criteria  and  associated  standards  inform  the 
instructor  and  student  as  to  what  is  to  be  trained  and  to  what  ievei.  Training  effectiveness 
evaiuations  (TEEs)  are  then  possibie.  The  use  of  simuiated  weapons  and  targets  has  greatiy 
decreased  the  cost  (and  increased  the  safety)  of  ACM  practice.  The  producer  of  the 
instrumented  air  combat  ranges  for  both  the  Navy  and  the  Air  Force  estimates  that  the 
TACTS/ ACMI  reduces  air  combat  training  costs  by  more  than  $100  miiiion  annuaiiy  (Cubic 
Corporation,  1978).  According  to  Biii  Doiiard  (1980),  the  TACTS  manager  at  Miramar,  the  first 
system  deveioped  for  Miramar  cost  approximateiy  $25  miiiion  in  R&D  and  instaiiation  of  the 
system  in  an  operationai  mode.  The  system  costs  iess  than  $1,000  an  hour  to  operate. 
However,  these  savings  must  be  assessed  in  terms  of  performance  effectiveness  measures  on 
the  TACTS.  In  view  of  the  above  described  iimitations  of  originai  TACTS  PMS,  the  recent  FPT 
message  seating  the  need  to  deveiop  a  PM  system  for  use  with  the  TACTS,  and  the 
tremendous  potentiai  vaiue  of  TACTS  performance  criteria,  it  is  ciear  that  functionai 
requirements  for  a  PM  system  shouid  be  identified.  Psychometric  Requirements:  objectivity, 
reiiabiiity,  vaiidity,  sensitivity,  and  quaiity  controi.  Training  Requirements:  evaiuative,  diagnostic, 
remediai,  timeiy,  affordabie,  adaptabie,  automated,  manageabie,  accessibie,  and  acceptabie. 


Entry  #6 

Training  Effectiveness  Evaiuation  of  Device  A/F37-A-T59 

1982 

Air  Force 

C-130 

Human  performance  data  reported 


Training  effectiveness  evaiuation  of  an  upgrade  to  C-130  simuiator,  with  addition  of  instrument 
fiight  simuiator  featuring  simuitaneous  training  of  piiot,  copiiot,  fiight  engineer,  and  navigator.  A 
biended  approach  of  ciassroom  and  simuiator  training  on  AF  crews  indicated  a  training 
effectiveness  ratio  of  .5  with  comparison  to  a  controi  group  not  received  the  simuiator  training. 
This  contributes  to  awareness  about  evaiuating  training  of  equipment  upgrades. 


Entry  #43 

Reiative  cost  and  training  effectiveness  of  the  F-111  6883  Converter/Fiight  Controi 

System 

1982 
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Air  Force 
F-1 1 1  aircraft 

Fluman  performance  data  reported 
Generaiizabie 

TEE  Study;  Knowiedge  Gap  .  TEE  experiment  with  miiitary  avionics  technicians  using  2D 
maintenance  trainer.  The  intent  of  the  study  was  to  examine  possibie  expanded  use  of 
simuiation  to  improve  training  vaiue  and  cost  effectiveness  of  maintenance  training.  Individuai 
maintenance  troubieshooting  skiii  and  procedurai  knowiedge  were  tested.  Cost  date,  covering  a 
15  year  iife  cycie  were  aiso  studied. 


Entry  #33 

Training  Effectiveness  Evauation  (TEE)  of  the  Advanced  Fire  Fighting  Training  System 

1983 

Navy 

Advanced  Fire  Fighting  Training  System 
Fluman  performance  data  reported 

Generaiizabie 

Effort  resuited  in  the  detaiied  specification  of  the  consoiidated  data  base  with  a  description  of 
steps  necessary  for  deveiopment  and  maintenance.  The  consoiidated  Human  Resource 
Technoiogies  to  encompass  maintenance  manpower,  instructionai  system  deveiopment,  job 
guide  deveiopment,  system  ownership  costs  and  human  resources  in  design  tradeoffs.  Report 
outiines  major  data  storage  for  aii  areas  identified  above. 


Entry  #20 

Resuits  of  the  Part-Task  Shiphandiing  Trainer  Pre-Prototype  Training  Effectiveness 
Evaiuation  (TEE) 

1985 

Navy 

0 

Human  performance  data  reported 
Generaiizabie 

The  Part  Task  Trainer  for  Shiphandiing  (PARTT-SHIP)  device  represents  one  portion  of  the 
Navai  Training  Equipment  Center's  shiphandiing  research  program.  This  device  was  functionai 
defined  after  conducting  a  Navy  shiphandiing  task  anaiysis,  and  deveioping  a  functionai 
specification  for  a  trainer  to  accompiish  approximateiy  80  percent  of  the  training  objectives 
required  for  basic  and  intermediate  ievei  shiphandiers.  The  PARTT-SHIP  is  a  pre-prototype  of  a 
part-task  shiphandiing  trainer.  The  pre-prototype  is  a  iow  cost  version  that  mimics  the  actuai 
capabiiities  of  a  fuiiy  capabie  shiphandiing  part-task  trainer.  The  goai  of  buiiding  a  pre-prototype 
was  to  demonstrate  and  evaiuate  in  severai  ways  the  capabiiity  of  the  actuai  device  which  couid 
be  buiit.  The  quantitative  portion  of  the  PARTT-SHIP  demonstration  was  recorded  for  Training 
Effectiveness  Evaiuation  purposes.  In  both  collision  avoidance  and  restricted  waters  training, 
experimental  results  showed  PARTT-SHIP  to  be  an  equally  effective  training  device  compared 
to  a  full  bridge  simulator.  Results  indicate  a  potential  for  PARTT-SHIP  as  part  of  a  total 
shiphandling  training  system. 


Entry  #32 

Literature  Review  -  Army  Training:  M16A1  Rifle,  TOW  &  DRAGON  Weapons  Systems 
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1986 

Army 

M16A1  Rifle,  TOW,  DRAGON 
Human  performance  data  reported 

Not  generalizable 

Effort  resulted  in  the  detailed  specification  of  the  consolidated  data  base  with  a  description  of 
steps  necessary  for  development  and  maintenance.  The  consolidated  Human  Resource 
Technologies  to  encompass  maintenance  manpower,  instructional  system  development,  job 
guide  development,  system  ownership  costs  and  human  resources  in  design  tradeoffs.  Report 
outlines  major  data  storage  for  all  areas  identified  above. 


Entry  #54 

On-The-Job  Training:  Development  and  Assessment  of  a  Methodology  for  Generating 
Task  Proficiency  Evaluation  Instruments 

1986 

Air  Force 

Generic  application 

Human  performance  data  reported 

Generalizable 

This  report  describes  the  development  and  assessment  of  a  methodology  for  generating  on-the- 
job  training  task  proficiency  assessment  instruments.  The  development  procedures  allow 
subject-matter  experts  without  experience  in  assessment  methodology,  to  construct  task 
proficiency  assessment  instruments  which  can  be  used  by  supervisors  to  assess  trainee 
proficiency  at  specific  tasks.  The  reliability  and  validity  of  the  procedures  and  of  the  forms 
resulting  from  procedural  application  were  examined  in  two  Air  Force  career  fields,  aircraft 
maintenance  and  security  police.  The  results  of  the  procedural  assessment  indicated  that  SMEs 
could  reliably  apply  the  development  procedures  to  construct  training  evaluation  forms  which 
accurately  and  completely  depict  the  critical  aspects  of  task  performance.  This  report  includes  a 
detailed  descriptions  of  the  assessment  methodologies  and  results.  Knowledge  Gap  input: 
The  ability  to  accurately  and  objectively  assess  an  individual's  level  of  performance  on  the  job  is 
important  to  major  systems  for  personnel  selection,  assignment,  training,  and  utilization.  This 
can  be  useful  in  Developmental  Testing,  offering  standardization  with  regard  to  the  construction 
of  assessment  instruments,  administration  of  performance  evaluations,  the  scoring  of  results, 
and  providing  feedback.  Implementation  Gap  input:  The  structured  technique  described  in  this 
report  for  assessing  performance,  on  the  job  or  during  development/operational  testing  can 
provide  structured  feedback  to  system  designers. 


Entry  #41 

F-16  Limited  field  of  view  simulator  training  effectiveness  study 

1987 

Air  Force 

F-16 

Human  performance  data  reported 
Generalizable 

TEE  Study;  Knowledge  Gap  .  Used  subjective  evaluation  (trainee  reaction)  to  limited  field  of 
view  simulation.  Military  pilots  completed  a  questionnaire  following  simulation  training.  Study 
focused  on  examining  the  effectiveness  of  limited  field  of  view. 
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Entry  #36 

MANPRINT  Findings  from  the  Investigative  Operationai  Assessment  of  the  Joint  Tacticai 
Information  Distribution  System  (JTIDS) 

1988 

Joint 

Joint  Tacticai  Information  Distribution  System  (JTIDS) 

Human  performance  data  reported 

Not  generalizable 

This  report  addressed  training  oniy  in  the  context  of  the  Investigative  Operationai  Assessment 
(lOA)  and  covered  training  to  test  operator  and  maintenance  training  for  the  specific  test  event. 


Entry  #49 

Human  Factors  Performance  Data  for  Future  Forward  Area  Air  Defense  Systems 
(FAADS) 

1989 

Army 

Forward  Area  Air  Defense  Systems  (FAADS) 

Human  performance  data  reported 

Generaiizabie 

Testifying  before  Congress  in  1977,  an  Army  officiai  stated  that  "We  expect  this  somewhat 
unorthodox  approach  to  permit  a  much  reduced  deveiopment  time,  thus  resuiting  in  an  eariiest 
fieiding  date,  aibeit  with  higher  but  acceptabie  risks.”  Despite  the  use  of  many  off  the  sheif 
technoiogies  that  were  intended  to  aiiow  rapid  and  iow-cost  deveiopment,  a  series  of  technicai 
probiems  and  massive  cost  overruns  resuited  in  the  canceiiation  of  the  Sergeant  York  project  in 
1985  by  the  Secretary  of  Defense.  Aithough  much  of  the  processing  of  the  coiiected  data 
was  aiso  canceiied  or  brought  to  a  hasty  conciusion,  an  extensive  foiiow-on  evaiuation  was 
conducted.  The  research  documented  the  anaiysis  of  training  and  human  factors- re iated  data 
from  the  Sgt  York  Foiiow-On  Evaiuation  I,  which  generated  objective  performance  measures  on 
individuai  soidier,  team,  and  system  performance.  Objective  performance  data  were  examined 
with  respect  to  fourteen  issues,  such  as  varying  target  workioads  on  the  operator  or  the  negative 
impact  of  some  semi-automation.  The  data  yieided  findings  on  many  issues  reievant  to  the 
design  of  a  future  area  air  defense  system.  The  report  provides  suggestions  on  appiying  such 
data  to  future  soidier-system  performance  modeis,  which  can  be  remarkabiy  heipfui  in  making 
decisions  about  future  FAADS.  Awareness  Gap  input:  Faiiures  of  canceiied  system  can  iead  to 
training  and  performance  improvements  in  simiiar,  foiiow  on  systems  if  properiy  anaiyzed. 
Impiementation  Gap  input:  Maintaining  detaiied  human  performance  data  during  DT/OT  aiiows 
secondary  anaiyses  for  other  system  designers. 


Entry  #52 

Embedded  Training  Software  Specifications  for  the  FOG-M  System  Demonstration 

1989 

Army 

FOG-M,  Fiber-Optic  Guided  Missiie 
Human  performance  data  reported 

Generaiizabie 
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This  report  describes  embedded  training  (ET)  concepts  investigated  for  the  FOG-M,  Fiber-Optic 
Guided  Missiie,  system  being  deveioped  by  the  U.S.  Army  Missiie  Command.  This  document 
presents  ET  software  specifications  for  a  demonstration  of  FOG-M.  Specifications  are  inciuded 
for  embedded  mission  simuiation,  computer-aided  instruction,  and  part-task  training  of  the  FOG- 
M  gunner.  The  structured  specifications  are  intended  for  use  by  FOG-M  programmers-who  wiii 
design  and  code  the  ET  software.  Knowiedge  Gap  input:  Detaiied  exampie  of  functions  and 
structures  to  embed  training  in  software  packages  of  a  weapon  system. 


Entry  #53 

Line  of  Sight-Forward  (Heavy)  Surrogate  Assessment 

1989 

Army 

Line  of  Sight-Forward 

Human  performance  data  reported 

Not  generaiizabie 

The  concept  feasibiiity  assessment  was  conducted  using  two  manufacturers'  prototype  or 
"surrogate"  weapon  systems  to  provide  baseiine  data  for  addressing  manpower,  personnei,  and 
training  issues.  A  front-end  anaiysis  identified  mission  and  performance  requirements.  The 
training  requirements  anaiysis  was  guided  by  the  TRADOC  systems  approach  to  training. 
Anaiyses  indicated  that  the  Army  can  support  such  a  system  with  respect  to  quaiity  of  personnei 
and  abiiity  to  train  these  personnei.  The  gunner's  engagement  task  requires  a  seven-step 
operationai  sequence  divided:  between  two  hand-operated  controi  devices  and  one  foot  switch, 
it  is  recommended  that  this  method  be  studied  to  determine  its  trainabiiity  and  impact  on  system 
effectiveness.  Knowiedge  Gap  input:  This  study  provides  an  exampie  of  how  to  examine 
training  requirements  in  a  front-end  anaiysis,  with  positive  resuits. 


Entry  #48 

Simuiation-Based  Assessment  of  Automated  Command,  Controi,  and  Communication 
Capabiiities  for  Armor  Crews  and  Piatoons:  The  intervehicuiar  information  System 

1991 

Army 

inter  Vehicuiar  information  System  (iViS) 

Human  performance  data  reported 

Generaiizabie 

The  intervehicuiar  information  System  (iViS)  is  an  automated  C3  dispiay  in  the  upgraded  Biock 
2  Ml  Abrams  tank.  This  simuiation-based  research  compared  the  performance  of  tank  crews 
and  tank  piatoons  using  a  prototype  iViS  with  the  performance  of  tank  crews  and  tank  piatoons 
using  conventionai  C3  and  navigationai  toois,  inciuding  a  radio,  paper  map,  protractor,  and 
compass.  Armor  crews  and  piatoons  equipped  with  iViS  significantiy  outperformed  the  crews 
and  piatoons  using  conventionai  techniques  on  every  composite  measure  evaiuated.  iViS- 
equipped  crews  and  piatoons  executed  a  C3  exercise  and  offensive  and  defensive  combat 
missions  faster,  reported  their  own  iocation  and  battiefieid  events  better,  and  successfuiiy 
executed  more  fragmentary  orders  than  crews  and  piatoon  equipped  with  conventionai  toois. 
Awareness  Gap  input:  The  findings  suggest  criticai  Armor  performance  and  training  impiications 
and  identified  25  design  issues  in  areas  such  as  report  queueing  and  routing,  position 
navigation,  and  icons.  These  can  be  generaiized  to  other  command,  controi,  and 
communication  upgrades  to  major  systems. 
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Entry  #37 

C3I  Test-Instrumentation  System:  MANPRINT  Evaluation  of  the  Data  Collection 

Subsystem 
1992 

Army 

C3I  Test-Instrumentation  System:  MANPRINT  Evaluation  of  the  Data  Collection 

Subsystem 

Human  performance  data  reported 
Not  generalizable 

There  was  no  formal  training  program  developed  or  planned  and  a  formal  training  evaluation 
was  not  conducted. 


Entry  #50 

Training  Requirements  Analysis  for  the  Combat  Vehicle  Command  and  Control  System 
Tactical  Operations  Center 
1992 
Army 

Close  Combat  Tactical  Trainer 
Human  performance  data  reported 

Generalizable 

This  research  describes  the  results  of  a  training  requirements  analysis  conducted  to  support 
development  of  the  Combat  Vehicle  Command  and  Control  System.  The  CVCC  is  a  set  of 
selected  futuristic  components  with  functions  simulated  in  the  Close  Combat  Test  Bed 
environment.  The  objective  of  the  task  analysis  was  to  (1)  provide  the  minimum  essential  task 
information  needed  to  support  the  early  assessment  of  CVCC  Tactical  Operations  Center 
training  requirements  and  (2)  assess  which  TOC  tasks  and  skills  should  be  included  in  future 
training  programs.  Knowledge  Gap  input:  Application  of  training  analysis  techniques  in  refining 
requirements  as  determined  by  user  experiences  in  simulated  environments. 


Entry  #51 

Hardmand  III:  A  Patriot  Growth  Application 

1992 

Army 

Patriot 

Human  performance  data  reported 
Generalizable 

Report  is  an  application  of  HARDMAN  III  methodology,  a  suite  of  software  tools  to  improve  the 
design  and  acquisition  process  for  hardware  and  software  systems  that  applies  mission 
simulation  to  examine  manpower,  personnel,  and  training  constraints.  It  was  applied  to  a 
proposed  improvement  (Navigation  Emplacement  System)  to  the  Patriot  Air  Defense  System. 
The  analyses  included  system  performance  requirements  and  projected  manpower  and 
personnel  characteristics  for  Patriot  occupational  specialties  in  the  fielding  years;  it  also  included 
workload  analyses  and  analyses  of  the  effects  of  personnel  quality  and  training  and 
environmental  stressors  on  mission  performance.  The  Patriot  Growth  Application  is  a  good 
example  of  the  concept,  methodology,  and  output  of  HARDMAN  III  analyses  through  user 
participation  on  an  actual  system.  Awareness  Gap  input:  The  HARDMAN  methodology  can  be 
applied  to  the  hardware/software  subsystem  requirements  of  major  acquisitions  from  a 
personnel  and  training  point  of  view. 
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Entry  #9 

An  Investigation  of  Integrated  Product  Development:  A  Case  Study  of  an  F-22  Prime 
Contractor 

1994 

Air  Force 
F-22 

Human  performance  data  reported 
Generalizable 

A  case  study  of  implementing  an  Integrated  Product  Team  structure  into  an  AF  System  Program 
Office  for  the  F-22.  A  Training  Systems  IPT  is  identified  on  p.  14,  but  no  other  explicit  mention 
of  system  training.  The  results  indicated  there  were  problems  with  the  implementation  of  the  IPT 
concept. 


Entry  #40 

Using  the  Backward  Transfer  Paradigm  to  Validate  the  63007A  AH-64  for  Simulator 
Training  Research  Advanced  Testbed  for  Aviation 
1994 
Army 
AH-64 

Human  performance  data  reported 
Generalizable 

TEE  Study-  knowledge  gap.  The  main  purpose  of  the  study  was  to  test  the  effectiveness  of  this 
simulator  in  training  tasks  required  for  later  more  advanced  simulator  and  aircraft  qualification 
skills.  The  backward  training  method,  at  the  time,  was  an  established  means  to  teach  landing 
and  hovering  tasks.  There  was  concern  that  the  visual  system  would  limit  effectiveness. 


Entry  #21 

User-Oriented  Design  Analysis  of  the  VESUB  Technology  Demonstration  System 
1997 
Navy 
1 

Human  performance  data  reported 
Generalizable 

This  study  provided  good  insights  into  design  of  training  systems  for  optimum  training  benefit.  It 
also  offered  five  general  areas  for  future  research  for  improved  lOS  designs. 


Entry  #26 

Formative  Evaluations  of  the  VESUB  Technology  Demonstration  System 

1997 

Navy 

VESUB  Technology  Demonstration  System 
Human  performance  data  reported 

Generalizable 
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Use  of  VR  provided  training  opportunities  that  are  not  avaiiabie  during  operations  with  the  actuai 
equipment  in  actuai  situations.  Training  that  wiii  expose  junior  officers  to  a  variety  of  chaiienging 
geographicai  and  environmentai  conditions  is  very  iimited  since  most  Commanding  Officers  put 
the  most  experienced  officers  on  watch  during  chaiienging  situations. 


Entry  #22 

Training  Effectiveness  Evaiuation  of  the  VESUB  Technoiogy  Demonstration  System  (U) 
1998 
Navy 
1 

Human  performance  data  reported 
Generaiizabie 

Use  of  this  technoiogy  provided  significant  performance  enhancements  over  current 
approaches,  and  shows  merit  for  inciusion  in  acquisition  programs.  However,  this  is  a  training 
system  deveiopment,  not  a  weapon  system.  This  report  provides  insights  into  how  weapon 
system  acquisition  programs  shouid  address  training  requirements  for  areas  iacking  sufficient 
detaii  or  rigor  to  adequateiy  addresses  operationai  training  requirements.  This  report  did  not 
address  cost  anaiysis,  impiementation  or  sustainment  of  the  system. 


Entry  #45 

Training  Effectivenes  Evaiuation  of  the  VSUB  technoiogy  demonstration  system 

1998 

Navy 

VSUB 

Human  performance  data  reported 
Generaiizabie 

Knowiedge  Gap  --  The  purpose  of  the  study  was  to  determine  the  effectiveness  of  a  specific 
trainer  for  submarine  personnei.  The  VSUB  trainer  fiiis  a  submarine  training  gap  because 
training  systems  existing  at  the  time  did  not  provide  detaiied  harbor  and  channei  ship  handiing 
training  for  the  officer  of  the  deck. 


Entry  #23 

VESUB  Technoiogy  Demonstration:  Project  Summary 
1999 
Navy 
1 

Human  performance  data  reported 
Generaiizabie 

Six  conciusions  and  recommendations  were  provided  to  support  future  improvements  in  VE 
training  system  deveiopment:  1)  VESUB  provides  effective  shiphandiing  training;  2)  team 
communications  is  a  must  in  the  deveiopment  of  compiex  systems;  3)  iegacy-based  system 
deveiopment  must  be  cioseiy  evaiuated;  4)  configuration  management  is  essentiai  for  success; 
5)  compiex  scenes  require  sufficient  fideiity  to  support  training  requirements;  and  6)  VE  offers 
unique  presentation  capabiiities  that  shouid  be  expioited. 


Entry  #35 
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Driver  Performance  Measurement  to  Support  Thermal  System  Development 
1999 
Army 
0 

Human  performance  data  reported 


This  short  report  described  a  human  factors  engineering  study  of  night  vision  enhancement  for 
Army  vehicle  drivers.  Two  systems,  Driver's  Vision  Enhancer  (DVE)  and  Night  Vision  Goggles 
(NVG)  were  used  to  detect  known  drop-offs  in  the  road  while  driving  a  HMMWV.  The  data  were 
not  reported.  A  related  study  of  non  responsive  pixels  found  found  no  significant  effect  of  display 
variations.  Training  was  never  discussed. 


Entry  #58 

Modeling  Human  Performance:  Impacting  System  Design,  Performance,  and  Cost 

2000 

Army 

Comanche 

Human  performance  data  reported 

Generalizable 

A  critical  component  to  be  considered  in  systems  acquisition-and  the  earlier  the  better-is 
human  performance.  In  this  report,  the  importance  of  using  human  performance  modeling  in 
order  to  impact  system  design,  performance,  and  cost  is  addressed.  The  examples  given  here 
are  all  based  on  models  developed  with  the  capabilities  present  in  IMPRINT  (the  Improved 
Performance  Research  Integration  Tool),  developed  by  the  Human  Research  and  Engineering 
Directorate  of  the  U.S.  Army  Research  Laboratory.  The  broad  use  of  models  and  simulations  is 
being  encouraged  in  the  military  at  this  time,  the  catch  phrase  being  “simulation-based 
acquisition”  (SBA).  The  answer  to  the  question  “When  should  human  performance  modeling  be 
used?”  is  "Early,  and  often  in  order  to  ascertain  the  basic  system  requirements,  the  precursor  to 
design,  and  the  role  of  the  human."  A  classic  example  from  the  U.S.  Army  concerns  what  is  now 
the  Comanche  helicopter.  It  was  first  envisioned  as  a  one-man  scout  and  attack  helicopter,  but 
early  assessments  of  expected  task  performance,  mental  workload,  and  reliance  on  automation 
proved  the  basic  concept  to  be  unworkable.  Today,  Comanche  is  being  built  as  a  two  man 
helicopter,  a  change  which  would  have  been  impossible  to  effect  later  in  the  acquisition  process 
without  a  complete  and  costly  re-design.  Another  example  of  the  application  of  IMPRINT'S  task 
network  modeling  capability  was  the  modeling  of  the  Land  Warrior  system  and  U.S.  Army’s 
Crusader  system,  originally  the  Advanced  Field  Artillery  System  (AFAS).  Knowledge  Gap  input: 
Example  of  human  performance  model  applied  to  several  systems.  Army  and  Navy,  resulting  in 
design  changes  with  impact  on  training. 

Entry  #39 

Aircrew  and  Maintainer  Warfighter  Performance  in  Aeronatuical  Systems  using  mission 
oriented  MOES 

2001 

Air  Force 

1 

Human  performance  data  reported 

Generalizable 
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Knowledge  Gap  .  Measurement  Method  Study:The  purpose  of  the  study  was  to  identify  and 
define  methods  for  incorporating  relevant  war  fighting  criteria  into  manpower,  personnel  and 
training.  Authors  state  that  R&D  program  descriptions  and  impact  statements  are  critical. 
Research  scientists  are  not  clear  on  what  the  criteria  are  or  where  or  how  to  get  measures.  The 
study  focused  on  identifying  material  on  MOEs/MOPs  important  to  AF  aircraft  -related  mission 
effectiveness. 


Entry  #28 

An  Assessment  of  the  Lead  Systems  Integrator  Concept  as  Applied  to  the  Future 
Combat  Systems  Program 
2006 
Army 

Army  Future  Combat  System  (FCS) 

Human  performance  data  reported 

Generalizable 

Effort  resulted  in  the  detailed  specification  of  the  consolidated  data  base  with  a  description  of 
steps  necessary  for  development  and  maintenance.  The  consolodated  Human  Resource 
Technologies  to  encompass  maintenance  manpower,  instructional  system  development,  job 
guide  development,  system  owenership  costs  and  human  resources  in  design  tradeoffs.  Report 
outlines  major  data  storage  for  all  areas  identified  above. 


Entry  #31 

Longbow  Stationary  Target  Indicator  Technical  Report 
2006 
Army 

AH-64D,  Apache  Longbow 
Human  performance  data  reported 

Generalizable 

Effort  resulted  in  the  detailed  specification  of  the  consolidated  data  base  with  a  description  of 
steps  necessary  for  development  and  maintenance.  The  consolodated  Human  Resource 
Technologies  to  encompass  maintenance  manpower,  instructional  system  development,  job 
guide  development,  system  owenership  costs  and  human  resources  in  design  tradeoffs.  Report 
outlines  major  data  storage  for  all  areas  identified  above. 


Entry  #38 

Looking  Back  at  20  Years  of  MANPRINT  on  Patriot:  Observations  and  Lessons 

2007 

Army 

Patriot  Missile  System 

Human  performance  data  reported 

Generalizable 

Training  issues  constituted  the  second  general  lesson  emerging  from  the  twenty  five  year 
evolution  of  the  Patriot  program.  System  enhancements  (new  hardware,  software, etc. 
progressively  made  the  system  more  complex.  However,  the  Increase  in  training  requirments 
was  not  matched  by  corresponding  improvements/changes  to  institutional  or  unit  training. 
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Entry  #56 

A  Littoral  Combat  Ship  Manpower  Analysis  using  the  Fleet  Response  Training  Plan 

2007 

N 

Littoral  Combat  Ship 

Human  performance  data  reported 

Generalizable 

The  training  requirements  for  each  sailor  assigned  to  the  LCS,  according  to  the  Preliminary 
Design  Interim  Requirements  Document,  will  use  a  human-centered  design  approach  to 
automate  decision  processes  that  optimize  manpower  workload.  This  approach  exploits 
SMARTSHIP  technologies  to  decrease  personnel  workload  without  compromising  the  design 
capability  to  fulfill  mission  requirements.  This  is  done  to  the  maximum  extent  practicable  in  order 
to  provide  on-demand  individual  and  team  training  mission  rehearsal  capability,  both  inport  and 
underway.  The  LCS  training  infrastructure  will  incorporate  a  responsive  and  flexible  training 
architecture  in  order  to  deliver  a  full  range  of  training  products  and  services  wherever  and 
whenever  required.  The  onboard  training  systems  will  span  the  training  continuum  from  warfare 
capabilities,  maintenance,  and  logistics  to  professional,  military,  leadership  and  personal 
enrichment  training.  This  approach  cultivates  the  total  sailor  and  supports  the  tenets  of  Sea 
Warrior.  Another  feature  of  the  training  system  is  its  availability  to  exercise  all  levels  of 
progressive,  basic,  intermediate,  and  advanced  training,  which  includes  team,  unit,  group,  joint, 
and  coalition  training  that  culminates  in  unrestricted  operations  certification.  A  mixture  of 
modeling  and  simulation  systems,  embedded  on-board  trainers,  web-enabled  or  PCbased 
distance  learning  systems,  and  netted  classroom  environments  will  be  installed  to  facilitate  this 
paradigm  shift  in  manpower  and  training  philosophies.  Manpower  requirements  are  based  on 
mission,  functions,  and  tasks  and/or  required  operational  capability  and  the  projected 
operational  environment.  A  ship’s  workload  is  determined  using  industrial  engineering 
techniques  to  yield  those  manpower  requirements.  The  Littoral  Combat  Ship’s  total  manpower 
requirements  have  been  set  at  75  and  plans  are  in  motion  to  train  the  first  Littoral  Combat  ship 
core  crew  prior  to  stepping  onboard.  Once  the  Littoral  Combat  Ship  is  commissioned  and  the 
core  crew  of  40  is  fully  prepared  to  take  the  deck,  it  will  enter  its  first  training  phase  commonly 
known  as  the  Unit  Level  Training  and  Readiness  Assessment.  This  research  centered  around  a 
40-member  crew  conducting  its  first  training  certification  using  completed  tasks,  fatigue  levels, 
and  workload  as  measures  of  performance  to  validate  its  manpower  requirements.  Crew 
certification  is  mandatory  for  all  new  construction  ships  in  which  an  emphasis  is  placed  on  the 
review  of  the  ship’s  overall  training  program,  the  ability  to  provide  a  minimum  number  of  qualified 
crewmembers  to  support  all  underway  requirements,  and  whether  all  training  objectives  are 
being  satisfied.  After  the  crew  certification  period,  core  crews  will  experience  an  evaluation 
within  the  initial  phase  of  training  called  the  Fleet  Response  Training  Plan  (FRTP).  This 
assessment  is  called  the  Unit  Level  Training  and  Readiness  Assessment  (ULTRA).  The  current 
challenge  is  whether  a  core  crew  of  40  can  effectively  complete  an  extensive  14-day  training 
assessment  within  acceptable  fatigue  levels. 


Entry  #44 

Development  and  Evaluation  of  a  virtual  terrain  table  for  night  vision  goggle  training 
2008 
Air  Force 
1 

Human  performance  data  reported 
Generalizable 
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TEE  Study;  Knowledge  Gap  .  The  purpose  of  the  study  was  to  evaluate  the  effectiveness  of  a 
virtual  terrain  board  (NVG  training  typically  uses  a  physical  terrain  board)  to  teach  NVG 
capabilities  and  limitations.  Study  focused  on  the  feasiblltly  of  replacing  the  physical  board  with  a 
virtual  simulation  of  same. 


Entry  #1 

Recognition  of  Combatants-Improvised  Explosive  Devices  (ROC-IED)  Training 
Effectiveness  Evaluation 
2009 

Army 

JIEDDO  sponsor 

Human  performance  data  reported 
Generallzable 

This  evaluation  assessed  the  effectiveness  of  the  Recognition  of  Combatants-Improvised 
Explosive  Devices  (ROC-IED)  computer-based  training  program  as  a  stand-alone  training  aid 
and  as  a  training  aid  that  can  supplement  the  training  given  by  the  unit’s  trainer.  The  evaluation 
employed  military  test  subjects  and  used  three  experimental  conditions  measuring  two  levels  of 
learning  outcomes.  The  results  Indicated  the  knowledge  level  of  Counter  lED  can  be  taught 
through  computer  based  training. 


Entry  #27 

Understanding  Demonstration-based  Training:  A  Definition,  Conceptual  Framework,  and 
Some  Initial  Guidelines 
2009 
Army 
1 

Human  performance  data  reported 
Generallzable 

This  report  explains  a  multi-level  framework  for  designing  training  that  offers,  based  on  empirical 
data  and  sound  theory,  a  systematic  and  more  affordable  approach  to  developing  the  full  range 
of  training  strategies  within  an  acquisition  program. 


Entry  #59 

Summary  Technical  Report  Development  of  Reaper  Mission  Essential  Competencies 

2009 

Air  Force 

Reaper  UAV 

Human  performance  data  reported 
Generallzable 
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This  document  summarizes  the  process  used  to  create  Mission  Essentiai  Competencies  (MECs 
SM)  for  Reaper,  it  presents  the  finai  MECs  and  supporting  competencies  identified  during  the 
MEC  process,  as  weii  as  the  associated  Reaper  knowiedge  and  skiiis,  iearning  environments, 
mission  types,  and  experiences.  MECs  are  higher-order  individuai,  team,  and  inter-team 
competencies  that  a  fuiiy  prepared  piiot,  operator,  crew  or  fiight  requires  for  successfui  mission 
compietion  under  adverse  conditions  in  a  non-permissive  environment.  Training  programs  for 
combat  aviators  have  historicaiiy  focused  on  in-fiight  training,  not  aiways  being  abie  to  repiicate 
the  dynamic  environment  encountered  during  wartime  operations.  The  inabiiity  to  reiiabiy  and 
safeiy  train  in  a  highiy  reaiistic  combat  environment  resuits  in  an  adaptation  period  for  aircrews 
during  the  initiai  stages  of  a  confiict.  This  adaptation  period  represents  a  gap  between  what 
couid  be  reaiisticaiiy  trained  and  what  is  expected  in  the  combat  arena.  Training  using 
Distributed  Mission  Operations  simuiations  overcomes  this  shortcoming.  The  Air  Force,  ACC 
has  a  methodoiogy  that  focuses  on  mission  execution  in  a  non-permissive  environment, 
centering  on  identifying  the  competencies  (or  MECs)  required  for  mission  compietion  during 
combat,  as  weii  as  the  knowiedge  and  skiiis,  supporting  competencies,  and  experiences 
required  for  MEC  proficiency. 


Entry  #60 
BW 

Summary  Technicai  Report  Deveiopment  of  Giobai  Hawk  Mission  Essentiai 
Competencies 
2009 

Air  Force 

Giobai  Hawk  UAV 

Human  performance  data  reported 

Generaiizabie 

This  document  summarizes  the  process  used  to  create  Mission  Essentiai  Competencies  (MECs 
SM)  for  Giobai  Hawk,  it  presents  the  finai  MECs  and  supporting  competencies  identified  during 
the  MEC  process,  as  weii  as  the  associated  Giobai  Hawk  knowiedge  and  skiiis,  iearning 
environments,  mission  types,  and  experiences.  MECs  are  higher-order  individuai,  team,  and 
inter-team  competencies  that  a  fuiiy  prepared  piiot,  operator,  crew  or  fiight  requires  for 
successfui  mission  compietion  under  adverse  conditions  in  a  non-permissive  environment. 
Training  programs  for  combat  aviators  have  historicaiiy  focused  on  in-fiight  training,  not  aiways 
being  abie  to  repiicate  the  dynamic  environment  encountered  during  wartime  operations.  The 
inabiiity  to  reiiabiy  and  safeiy  train  in  a  highiy  reaiistic  combat  environment  resuits  in  an 
adaptation  period  for  aircrews  during  the  initiai  stages  of  a  confiict.  This  adaptation  period 
represents  a  gap  between  what  couid  be  reaiisticaiiy  trained  and  what  is  expected  in  the  combat 
arena.  Training  using  Distributed  Mission  Operations  simuiations  overcomes  this  shortcoming. 
The  Air  Force,  ACC  has  a  methodoiogy  that  focuses  on  mission  execution  in  a  non-permissive 
environment,  centering  on  identifying  the  competencies  (or  MECs)  required  for  mission 
compietion  during  combat,  as  weii  as  the  knowiedge  and  skiiis,  supporting  competencies,  and 
experiences  required  for  MEC  proficiency.  Six  MECs  are  identified  for  the  Giobai  Hawk:  Pian 
and  Prepare  for  Mission,  Conduct  Aircraft  Launch/  Recovery  Sequence,  Conduct  Aircraft 
Handoff/  Change-over,  Coiiect  Targets,  Conduct  Post  Mission  Review,  Manage  Crew  Workioad. 


Entry  #61 

Summary  Technicai  Report  Deveiopment  of  Predator  Mission  Essentiai  Competencies 

2009 

Air  Force 

Predator  UAV 
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Human  performance  data  reported 
Generalizable 

This  document  summarizes  the  process  used  to  create  Mission  Essentiai  Competencies  (MECs 
SM)  for  Predator,  it  presents  the  finai  MECs  and  supporting  competencies  identified  during  the 
MEC  process,  as  weii  as  the  associated  Predator  knowiedge  and  skiiis,  iearning  environments, 
mission  types,  and  experiences.  MECs  are  higher-order  individuai,  team,  and  inter-team 
competencies  that  a  fuiiy  prepared  piiot,  operator,  crew  or  fiight  requires  for  successfui  mission 
compietion  under  adverse  conditions  in  a  non-permissive  environment.  Training  programs  for 
combat  aviators  have  historicaiiy  focused  on  in-fiight  training,  not  aiways  being  abie  to  repiicate 
the  dynamic  environment  encountered  during  wartime  operations.  The  inabiiity  to  reiiabiy  and 
safeiy  train  in  a  highiy  reaiistic  combat  environment  resuits  in  an  adaptation  period  for  aircrews 
during  the  initiai  stages  of  a  confiict.  This  adaptation  period  represents  a  gap  between  what 
couid  be  reaiisticaiiy  trained  and  what  is  expected  in  the  combat  arena.  Training  using 
Distributed  Mission  Operations  simuiations  overcomes  this  shortcoming.  The  Air  Force,  ACC 
has  a  methodoiogy  that  focuses  on  mission  execution  in  a  non-permissive  environment, 
centering  on  identifying  the  competencies  (or  MECs)  required  for  mission  compietion  during 
combat,  as  weii  as  the  knowiedge  and  skiiis,  supporting  competencies,  and  experiences 
required  for  MEC  proficiency.  Six  MECs  are  identified  for  the  Predator;  Pian  and  Prepare  for 
Mission,  Conduct  Aircraft  Handoffs,  Prepare  for  Mission  Execution,  Coiiect  Targets,  Empioy  Air- 
Ground  Weapons,  Conduct  Post  Mission  Review,  Manage  Crew  Workioad,  Conduct  MAC 
Operations,  Launch  and  Recovery. 


Entry  #8 

Cyber  Operations  Virtuai  Environments 
2010 
Air  Force 
1 

Human  performance  data  reported 
Generaiizabie 

The  report  presented  a  review  of  appiicabie  Science  and  Technoiogy  issues  for  the  design  of 
instruction  reiated  to  preparing  airmen  to  defend  against  and  respond  to  cyber  threats  to  their 
networks.  The  report  provided  a  notionai  framework  for  a  testbed  offering  muitipie  opportunities 
for  future  expansion  to  other  piatforms  and  a  rich  environment  in  which  to  expiore  the  effects  of 
cyber  attacks.  The  report  provides  an  awareness  for  the  design  of  future  training  systems. 
Reiated  to  NetCentric  functionai  capabiiity. 


Entry  #24 

KC-135  Simuiator  Systems  Engineering  Case  Study 
2010 
Air  Force 

KC-135  Aircrew  Training  System  (ATS) 

Human  performance  data  reported 

Generaiizabie 
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Report  states  that  simulators  can  provide  more  indepth  training  than  can  be  accomplished  in 
airplanes  and  provide  a  very  high  transfer  of  learning  and  behavior  from  the  simulator  to  the 
airplane.  The  use  of  simulators  has  resulted  in  safer  flight  training  and  cost  reductions  for  the 
operators  as  well  as  improvided  fuel  conservation  and  reduction  in  adverse  environmental 
effects.  The  KC-135  training  systems  is  managed  and  funded  as  a  separate  but  related 
acquisition  program.  Major  modifications,  particularly  to  the  Operational  Flight  Trainer  (OFT), 
have  been  successfully  planned  for,  budgeted,  and  implemented  over  the  past  17  years.  There 
are  several  reasons  on  why  this  teaming  relationship  has  succeeded.  Typically  with  acquisition 
programs  the  program  manager  or  chief  engineer  charged  with  the  development  program  would 
chair  major  reviews  (like  the  KC-135  ATS  System  Review  Board  [SRB])  with  the  using  command 
(in  this  case  AMC)  providing  a  briefing  of  their  issues  and  concerns  at  the  SRB.  Flowever,  the 
arrangement  that  has  evolved  for  KC-135  ATS,  which  has  proven  to  be  very  effective,  is  that  the 
AMC  manager  co-chairs  the  SRB  forum.  AMC,  the  ATS  Program  Office,  and  the  prime 
contractor(s)  draw  on  support  as  needed  to  ensure  proper  staffing  is  available  for  program 
execution.  Another  reason  for  the  team’s  success  is  their  ability  to  be  flexible  and  react  quickly 
to  customer  needs.  The  team  also  utilizes  a  requirement  verification  and  prioritization  review 
board  (called  the  SPRR  System  Priority  Requirements  Review)  that,  in  addition  to  upgrades 
driven  by  weapon  system  changes,  addresses  sustainment  related  hardware  and  software 
deficiencies/upgrades  required  to  improve  flight  simulator  fidelity.  Prioritization  reviews  include 
representatives  from  the  KC-135  Program  Office,  Ogden,  Contractors  (aircraft  and  simulator), 
and  user.  Part  of  the  reason  for  the  effectiveness  of  the  KC-135  ATS  upgrade  program  is  the 
simulator  team  has,  over  a  period  of  17  years,  evolved  into  a  very  effective  organization.  One  of 
the  challenges  facing  the  Government  in  2010,  when  the  current  contract  is  recompeted,  will  be 
to  foster  the  advantages  associated  with  long-term  support  contracts  (i.e.,  workforce  continuity, 
knowledgeable  support  personnel,  program  stability,  sense  of  ownership,  incentives  for  process 
improvements,  incentive  for  long-range  planning)  while  meeting  the  government’s  requirements 
for  increased  competition  and  shorter  term  contracts. 


Entry  #25 

Systems  Engineering  Applied  Leading  Indicators 
2010 
Navy 
0 

Human  performance  data  reported 
Generalizable 

Offered  a  systems  engineering  methodology  for  development  of  leading  indicators  for  more 
effective  program  decisions  based  on  more  than  just  cost  and  schedule.  But,  did  not  address 
human  performance  or  training  metrics.  The  report  also  cited  the  challenges  with  consistent 
availability  of  program  data  for  this  methodology.  Lastly,  the  report  suggested  maturation  to 
address  total  ownership  cost  control. 


Entry  #34 

Key  Human  System  Integration  Plan  Elements  for  Command  &  Control  Acquisition 
2010 
Navy 
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Human  performance  data  reported 
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This  paper  from  SPAWAR  describes  how  HSi  contributes  to  the  Acquisition  of  Command  & 
Controi  systems.  Emphasis  is  given  to  Human  Factors  Engineering  issues  and  HSi  processes 
and  reports.  Training  is  iisted  as  a  domain  of  HSi  and  the  need  for  a  Totai  Systeme  Training 
Architecture  (TSTA)  is  noted  aiong  with  a  Navy  Training  System  Pian  (NTSP). 


GAO  CONSEQUENTIAL  ABSTRACTS 


Entry  #501 

Achieving  B-2A  Bomber  Operationai  Requirements 

1997 

Air  Force 

B-2 

NSiAD-99-97 

The  report  states  that  poor  iow  observabie  reiiabiiity  and  iengthy  maintenance  times  reduce 
mission  capabie  rates,  adverseiy  affect  aircraft  avaiiabiiity  for  crew  training,  restrict  operations 
when  depioyed,  and  iimit  sustained  combat  operations. 


Entry  #502 

Evaiuation  of  Navy's  Anti-Submarine  Warfare  Assessment 

1999 

Navy 

ASW 

NSiAD-99-85 

The  assessment  conciuded  that  the  ASW  program  of  record  as  contained  in  the  proposed 
President’s  fiscai  year  1999  budget  was  adequate  to  meet  the  iikeiy  future  threat  and  identified 
the  foiiowing  near-,  mid-,  and  iong-term  ASW  mission  requirements.  Near  term 

requirements:  improve  ASW  crew  proficiency  by  increasing  training. 


Entry  #503 

Missiie  Risk  Reduction  Underway  But  System  Production  Pians  Need  to  be  Reexamined 

1998 

Joint 

AiM-9X,  Sidewinder 
NSiAD-98-45 

in  commenting  on  a  draft  of  this  report,  DOD  stated  that  the  projected  range  and  sensor  tracking 
capabiiity  of  AiM-9X  without  the  heimet-mounted  cueing  system  is  equivaient  to  the  capabiiity  of 
the  AA-1 1  threat  missiie  in  azimuth  and  exceeds  the  capabiiity  of  the  AA-11  in  range.  DOD’s 
position  is  based  on  using  the  fighter  aircraft  radar  to  cue  the  AiM-9X  missiie  to  the  target  of 
interest  when  it  is  beyond  the  view  of  the  aircraft’s  heads-up  dispiay.  Using  the  radar  to  cue  the 
missiie,  however,  wiii  take  more  time  and  be  iess  certain  than  with  the  heimet  and  wiii  require 
DOD  to  train  piiots  in  yet  to  be  deveioped  procedures  and  tactics  that  wouid  be  considerabiy 
different  than  current  practices  for  aeriai  combat. 
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Entry  #504 

Advanced  Concept  Technology  Demonstration 

1998 

Air  Force 

Predator 

NSIAD-99-4 

In  the  early  operational  assessment  of  the  Predator’s  ACTD  demonstration,  the  Director, 
Operational  Test  and  Evaluation,  did  not  make  a  determination  of  the  system’s  potential 
operational  effectiveness  or  suitability.  However,  the  system  was  found  to  be  deficient  in  several 
areas,  including  mission  reliability,  documentation,  and  pilot  training.  The  assessment  also 
noted  that  the  ACTD  demonstration  was  not  designed  to  evaluate  several  other  areas  such  as 
system  survivability,  supportability,  target  location  accuracy,  training,  and  staffing  requirements. 


Entry  #505 

Acquisition  Plans  for  Training  Aircraft  Should  Be  Reevaluated 

1997 

General 


NSIAD-97-172 

DOD  is  proceeding  with  plans  to  procure  a  fleet  of  JPATS  aircraft  that  may  exceed  the  quantity 
needed  to  meet  training  requirements.  Until  inconsistencies  in  the  data  used  to  calculate 
JPATS  requirements  are  resolved,  it  is  unclear  how  many  aircraft  should  be  procured. 
Furthermore,  DOD’s  schedule  for  procuring  the  aircraft  does  not  take  advantage  of  the  most 
economical  approach  that  would  allow  it  to  save  money  and  permit  more  time  for  operational 
testing. 


Entry  #506 

Acquisition  Cost  Estimates 

1993 

Air  Force 

B-2 


The  estimated  support  costs,  which  included  those  of  data,  training,  peculiar  support 
equipment,  and  software  support,  were  not  based  on  a  comprehensive  analysis  by  the  Air 
Force.  Justification  provided  by  the  Air  Force  for  the  costs  was  incomplete,  based  on  other 
weapon  systems,  or  preliminary  in  nature.  B-2  program  officials  stated  that  the  cost  estimate  for 
support  of  the  B-2  is  not  firm  because  the  overall  support  concept  for  the  20  aircraft  program  has 
not  yet  been  decided.  They  stated  that  a  more  accurate  estimate  of  support  cost  will  be  available 
when  the  1994  President’s  budget  is  announced.  The  peculiar  support  cost  element  includes 
$1,266  billion  associated  with  the  support  of  the  production  aircraft  and  simulators,  procurement 
of  technical  data,  and  procurement  of  items  for  maintenance  training  equipment. 


Entry  #507 

Selected  Aspects  of  the  Army's  Forward  Area  Air  Defense  System 
1990 
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Army 

FAADS 

NSIAD-90-191 

The  DIVAD  program,  however,  was  terminated  by  the  Secretary  of  Defense  in  August  1985 
because  it  couid  not  handie  the  stand-off  attack  heiicopter  threat.  Army  anaiysis  and  training 
had  reveaied  that  the  projected  air  threat  couid  strike  ground  forces  from  distances  beyond  the 
range  of  current  frontiine  missiie  systems  and  the  DIVAD. 


Entry  #508 

Acquisition  Issues  Facing  the  Army  Battie  Command,  Brigade  and  Beiow  Program 

1998 

Army 

FBCB2 

NSIAD-98-140 

The  oversight  effort  was  conducted  in  partnership  with  the  Army’s  Operationai  Test  and 
Evaiuation  Command,  in  recognition  of  the  unique  nature  of  the  experiment  (as  distinct  from  an 
operationai  test).  The  Director’s  report  aiso  identified  a  iack  of  (1)  adequate  digitai  connectivity; 
(2)  maturity  of  the  Appiique  and  the  Tacticai  Internet;  (3)  adequate  tactics,  techniques,  and 
procedures  for  operations  with  digitai  equipment;  and  (4)  tacticai  skiiis  resuiting  from  inadequate 
unit  coiiective  training. 


Entry  #509 

Progress  of  the  F-22  and  F/A-18E/F  Engineering  and  Manufacturing  Deveiopment 
Programs 

1999 

Air  Force 

F-22 

T-NSIAD-99-113 

Our  review  showed  that  the  Navy’s  statements  about  the  performance  of  the  E/F  reflect  the 
performance  of  the  Emodei  aircraft,  not  the  iess  capabie  F  modei.  Aiso,  the  statements  reflect 
the  projected  aircraft  performance,  not  the  actuai  performance  being  demonstrated  in  fiight  tests. 
Specificaiiy,  the  Navy.s  performance  vaiues  inciude  anticipated,  but  not  yet  demonstrated,  range 
improvements.  If  these  values  are  not  included  in  the  performance  estimates,  the  F  model 
aircraft  will  be  33  nautical  miles  short  of  meeting  its  interdiction  range  requirement.  This  is 
significant  because  (1)  the  F  model,  which  was  originally  planned  to  be  used  as  a  trainer  aircraft 
and  therefore  made  up  only  about  20  percent  of  the  total  buy,  now  comprises  about  56  percent 
of  the  total  buy  and  (2)  increased  range  over  the  current  C/D  aircraft  was  critical  to  justifying  the 
decision  to  buy  the  F/A-18E/F. 


Entry  #510 

Progress  Made,  but  Significant  Challenges  Remain 

1998 

Joint 


JCCS 

NSIAD/AIMD-98-257 
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DOD  has  experienced  a  mixture  of  successes  and  probiems  in  impiementing  Giobai  Command 
and  Controi  System.  For  exampie,  users  iike  some  of  the  additionai  features  it  provides 
compared  to  the  oid  system  and  found  them  productive.  These  features  inciude  mission-reiated 
communications  by  e-maii,  internet-iike  web  pages,  and  on-iine  discussion  groups.  Users  aiso 
iike  the  idea  of  being  provided  situationai  awareness  of  the  battiefieid.  Flowever,  some  key 
capabiiities,  such  as  the  system’s  operationai  pianning  function  and  the  situationai  awareness 
function,  have  experienced  probiems  and  are  performing  iess  effectiveiy  than  expected.  Aiso, 
operator  training  is  deficient,  data  exchange  procedures  with  coaiition  partner  have  not  been 
defined,  and  the  system  is  at  risk  of  faiiure  because  year  2000  probiems  have  not  been  fuiiy 
resoived. 


Entry  #51 1 

Successfui  Appiication  to  Weapon  Acquisitions  Requires  Changes  in  DOD's 
Environment 
1998 

Army 

Javeiin 

NSIAD-98-56 

The  Javeiin  is  designed  to  significantiy  improve  the  Army's  antitank  capabiiity.  This  capabiiity  is 
iimited  by  the  weather,  the  gunner's  distance  from  the  target,  and  the  gunner's  training. 
Flowever,  there  are  some  questions  as  to  whether  training  wiii  be  adequate  to  prepare  the 
soidier  to  (1)  acquire  vaiid  targets  and  (2)  identify  friend  from  foe  without  additionai  equipment, 
which  is  not  yet  deveioped.  We  discussed  the  Javeiin's  abiiity  to  identify  friend  from  foe  with 
officiais  from  the  Office  of  the  Deputy  Chief  of  Staff  for  Operations  and  Pians,  Washington  D.C.; 
and  the  Javeiin  Project  Office.  We  aiso  discussed  the  training  aspects  of  this  issue  and 
reviewed  documents  from  the  U.S.  Army  Infantry  School,  Fort  Benning,  Georgia;  Center  for 
Night  Vision  and  Electra-Optics,  Fort  Belvoir,  Virginia;  and  the  Javelin  Project  Office.  Training 
officials  at  the  U.S.  Army  Infantry  School  consider  the  currently  planned  Javelin  gunner  training 
inadequate  in  the  areas  of  recognition  and  identification.  In  their  opinion,  the  training  will  not 
adequately  teach  gunners  to  use  the  Javelin's  infrared  optics  to  recognize  tanks  from  wheeled 
vehicles  or  identify  friendly  tanks  from  enemy  tanks.  A  Javelin  logistics  official  said  the  Javelin 
training  will  teach  gunners  to  use  the  system--that  is,  turn  it  on,  lock-on  to  targets,  and  fire  the 
missile.  In  addition,  an  Army  Infantry  School  official  said  gunners  will  be  taught  general  thermal 
recognition  and  identification.  However,  both  said  the  gunners  will  not  be  taught  to  recognize  or 
identify  thermal  images  as  seen  through  the  Javelin  optics.  According  to  an  Army  research 
psychologist,  the  current  training  software  will  not  teach  gunners  to  recognize  (1)  moving 
targets,  (2)  targets  with  approximately  the  same  temperature  as  that  of  their  background,  or  (3) 
targets  whose  image  is  degraded  by  electronic  noise  -as  the  Javelin's  images  frequently  are. 


Entry  #512 

Javelin  Is  Not  Ready  for  Multiyear  Procurement 

1999 

Army 

Javelin 

NSIAD-96-199 


68 


Javelin’s  Chief  of  Logistics  said  the  Army  cannot  afford  to  maintain  both  the  low-rate  production 
and  redesigned  launch  unit  configurations.  He  said  that  if  soldiers  were  given  different  launch 
units,  the  Army  would  have  to  maintain  inventory  and  train  personnel  to  repair  both 
configurations.  In  addition,  the  Army  would  have  to  develop  and  produce  test  equipment  for  the 
low-rate  production  configuration  because  it  will  not  have  built-in-test  equipment  to  diagnose 
system  failures. 


Entry  #513 

Accelerated  Joint  STARS  Ground  Station  Acquisition  Strategy  Is  Risky 

1996 

Army 

JSTARS 

NSIAD-96-71 

In  some  instances,  problems  were  attributed  to  shortfalls  in  operator  training  or  another  non¬ 
materiel  cause.  The  majority  of  deficiencies  involved  software  fixes,  not  major  hardware 
redesign.  The  Army  has  also  gained  experience  operating  the  GSMs  assigned  to  the  III  Corps 
and  XVIII  Airborne  Corps  and  in  training  and  preparation  for  multi-service  OT&E.  In  November 
1995,  the  Program  Executive  Officer  for  Joint  STARS  certified  the  system  ready  for  OT&E, 
which  attests  to  the  developer’s  confidence  in  system  maturity. 


Entry  #514 

Concerns  With  Milstar's  Support  to  Strategic  and  Tactical  Forces 

1998 

Army 

MILSTAR 

NSIAD-99-2 

This  upgrade  effort,  while  compatible  with  the  Army’s  terminal  testing  schedule,  will  require  the 
Army  to  reconfigure  its  equipment  and  software  and  retrain  its  forces  when  the  automated 
communications  management  system  becomes  available  at  a  future  date.  Military  Strategic  and 
Tactical  Relay  (AF  manages). 


Entry  #515 

Evaluation  of  the  Navy's  1999  Naval  Surface  Fire  Support  Assessment 

1999 

Navy 

Navy  Surface  Fire  Support 
NSIAD-99-225 

The  estimated  cost  cited  for  reactivating  the  U.S.S.  Wisconsin  was  $209.4  million  and  for  the 
U.S.S.  Iowa,  was  $221.3  million,  including  repair  of  the  damaged  turret.  To  accomplish  the 
reactivation,  the  Navy  estimated  14  months  for  industrial  support  and  3  to  6  months  for 
modernization  and  training  on  and  certification  of  newly  installed  equipment. 


Entry  #516 

Navy's  Aviation  Logistics  System  Not  Ready  for  Deployment 

1990 

Navy 
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Aviation  Logistics 
IMTEC-90-11 

Additionaiiy,  the  program  manager  toid  us  that  existing  documentation  wiii  be  augmented  by 
word-of-mouth  and  on-the-job  training.  In  our  opinion,  this  approach  is  not  adequate  for  a 
system  as  iarge  and  important  as  NALCOMIS.  Aithough  system  maintenance  may  be  possibie 
without  a  manuai,  it  wiii  undoubtediy  be  more  difficuit  and  costiy  than  necessary. 


Entry  #517 

A  Rare  Opportunity  for  Lasting  Change 

1992 

Generai 


NSIAD-93-15 

The  desire  of  program  sponsors  to  keep  cost  estimates  as  iow  as  possibie  and  to  present 
attractive  miiestone  scheduies  has  encouraged  the  use  of  unreasonabie  assumptions  about  the 
pace  and  magnitude  of  the  technicai  effort,  materiai  costs,  production  rates,  savings  from 
competition,  and  other  factors.  In  some  cases,  acquisition  cost  estimates  have  been  kept  iow  by 
exciuding  reievant  program  costs-such  as  the  cost  of  training  equipment-which  shouid  be 
inciuded  in  program  cost  estimates.  Successfui  performance  of  missions,  assuming  weii 

trained  operators  and  reiiabie  equipment,  requires  a  weapon  system  that  is  operationaiiy 
effective.  “Effectiveness”  generaiiy  refers  to  the  abiiity  of  a  weapon  to  successfuiiy  engage  the 
enemy.  For  a  majority  of  weapons,  that  means  the  abiiity  to  reach  the  target  area,  find  targets, 
and  destroy  or  disabie  them  with  munitions  such  as  missiies  or  torpedoes.  In  a  1983 

classified  report  on  the  Army’s  Patriot  air  defense  system,  we  reported  that  the  diagnostic 
software  used  with  the  system’s  built-in  test  equipment  had  successfully  identified  faulty 
components  only  50  to  60  percent  of  the  time.  To  address  these  problems,  the  Army  upgraded 
the  software,  added  another  level  of  maintenance,  and  increased  maintenance  training.  Initial 
investment  costs  for  these  changes  was  about  $94  million. 


Entry  #518 

Status  of  Selected  Programs 

1990 

General 


NSIAD-90-159 

The  initial  operational  evaluation  tests  were  conducted  from  November  1988  to  September 
1989.  The  tests  included  (1)  captive  flight  tests  to  armor  targets,  (2)  a  force  development  test 
and  experiments  to  investigate  tactics,  training,  and  crew  performance,  and  (3)  actual  missile 
firings  to  demonstrate  capability  against  helicopters  and  armored  vehicles. 


Entry  #519 

The  Special  Operations  Forces  Aircrew  Training  System  at  One  Year 

1992 

Air  Force 
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NSIAD-92-52 

There  has  been  a  2-month  schedule  slippage  because  problems  in  preliminary  software  design, 
development  of  in-class  training  curricula  and  materials,  and  cockpit  configuration  changes 
require  more  work  than  anticipated.  The  prime  contractor  is  working  with  the  subcontractors  to 
develop  and  implement  a  new  schedule  for  some  tasks  to  meet  all  major  milestones. 


Entry  #520 

Commercial  Components  Used  Extensively  in  Tactical  Trucks 

1994 

Army 


NSIAD-94-240 

The  Army  requires  that  its  vehicles  be  fielded  with  detailed  technical  manuals  and  other 
documentation  covering  the  operation  and  maintenance  of  the  vehicle.  The  Army  requires 
specialized  manuals  because  (1)  Army  drivers  and  mechanics  generally  are  not  as  well  trained 
as  commercial  drivers  and  mechanics. 


Entry  #521 

Defense  Weapons  Systems  Acquisition 

1997 

General 


HR-93-7 

The  desire  of  program  sponsors  to  keep  cost  estimates  as  low  as  possible  and  to  present 
attractive  milestone  schedules  has  encouraged  the  use  of  unreasonable  assumptions  about  the 
pace  and  magnitude  of  the  technical  effort,  material  costs,  production  rates,  savings  from 
competition,  and  other  factors.  In  some  cases,  acquisition  cost  estimates  have  been  kept  low  by 
excluding  relevant  program  costs-such  as  the  cost  of  training  equipment-which  should  be 
included  in  program  cost  estimates. 


Entry  #522 

Issues  Concerning  Acquisition  of  THAAD  Prototype  System 

1996 

Army 

THAAD 

NSIAD-96-136 

The  remaining  30  UOES  interceptors  were  to  be  used  in  operational  suitability  tests  that  did  not 
involve  firing  interceptors.  However,  the  purposes  of  those  tests  can  be  accomplished  with 
training  rounds  or  with  the  backup  demonstration  and  validation  phase  interceptors.  The 
Army  also  plans  to  conduct  a  7-week  limited  user  test  after  completing  the  14  demonstration  and 
validation  flight  tests.  The  limited  user  test  will  build  on  the  results  of  the  demonstration  and 
validation  flight  tests  and  focus  on  operational  effectiveness  and  suitability  issues,  including 
mission  performance  and  system  supportability.  The  primary  events  to  be  conducted  during  the 
limited  user  test  will  be  field  training  exercises,  command  post  exercises,  modeling  and 
simulations,  and  a  flight  test  using  an  interceptor  already  under  contract.  Results  from  the  limited 
user  test  are  not  scheduled  to  be  available  until  early  1 998. 
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Entry  #523 

THAAD  Restructure  Addresses  Problems  But  Limits  Early  Capability 

1999 

Army 

THAAD 

NSIAD-99-142 

To  identify  underlying  reasons  for  the  program’s  difficulties,  we  reviewed  pertinent  government 
and  contractor  documentation,  including  contract  files,  audit  reports,  schedules,  briefings,  cost 
reports,  integrated  product  team  minutes,  and  contractor  resolution  plans  and  training  plans. 
We  also  reviewed  independent  studies  and  discussed  the  studies’  findings  with  knowledgeable 
officials.  We  compared  the  results  of  our  review  to  the  findings  of  the  independent  studies. 


Entry  #524 

Comprehensive  Strategy  Needed  to  Improve  Ship  Cruise  Missile  Defense 
2000 
Navy 


GAO/NSIAD-00-149 

Impediments  to  Availability:  SLQ-32  Electronic  Warfare  System  V2  0.72  0.70  0.69  High  cost  of 
parts,  lack  of  funding  for  parts  and  upgrades,  parts  failure,  lack  of  technician  experience,  and 
training;  SPS-48E  Radar  System  0.73  0.81  0.77  Inadequate  training  of  shipboard  technicians, 
lack  of  technical  schematics 


Entry  #525 

Actions  Needed  to  Enhance  Success  of  Reengineering  Initiatives 

2000 

Joint 


GAO/NSIAD-00-89 

Logistics  support  personnel  from  the  Joint  Chiefs  of  Staff  and  combat  commands  in  the  United 
States  and  Europe  voiced  a  number  of  concerns  about  the  potential  effects  that  some 
reengineering  efforts  could  have  on  their  operational  capability.  These  concerns  involved  the 
presence  of  increasing  numbers  of  contractor  personnel  on  or  near  the  battlefield,  the  ability  of 
contractors  to  meet  “surge  requirements”,  the  potential  reduction  of  rotational  positions  to  meet 
training  requirements 


Entry  #526 

Chemical  and  Biological  Defense:  Units  Better  Equipped,  but  Training  and  Readiness 
Reporting  Problems  Remain 

2000 

Joint 


GAO-01-27 
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Many  units  were  inadequately  trained  in  basic  tasks  critical  to  surviving  and  operating  in  a 
chemical  or  biological  environment.  Our  current  and  prior  work  as  well  as  the  work  of  DOD’s 
Office  of  the  Inspector  General  found  that  commanders  were  not  integrating  chemical  and 
biological  defense  into  unit  exercises  and  the  training  was  not  always  realistic  in  terms  of  how 
units  would  operate  in  war.  For  example,  we  were  told  by  Marine  Corps  officers  responsible  for 
chemical  and  biological  defense  training  at  the  unit  we  reviewed  that  commanders  are  not  fully 
integrating  chemical  and  biological  defense  into  unit  exercises,  as  required  by  Marine  Corps 
policies,  because  operating  in  protective  equipment  is  difficult  and  time  consuming  In 
September  1999,  we  reported  that  the  Army’s  combat  training  centers  were  restricting  the 
simulated  use  of  chemical  weapons  against  the  units  being  trained  because  units  were  arriving 
at  the  centers  with  lower  levels  of  proficiency  in  chemical  and  biological  defense  than  in  the  past 
Officials  at  the  units  we  reviewed  stated  that  chemical  and  biological  defense  training  is  being 
adversely  impacted  by  (1)  a  shortage  of  chemical  and  biological  defense  specialists  and  (2) 
specialists  being  assigned  multiple  responsibilities  unrelated  to  their  specialties 


Entry  #527 

Actions  to  Improve  Navy  SPAWAR  Low-Rate  Initial  Production  Decisions 


Navy 
Info  Tech 
GAO-01-735 

A  suitability  problem  involves  a  system  not  satisfactorily  meeting  one  or  more  requirements, 
including  reliability,  maintainability,  logistics  support,  or  training.  These  problems  may  delay 
progress  in  achieving  the  Navy’s  vision  for  using  information  technology  to  attain  and  maintain 
network-centric  warfighting  knowledge  and  decision-making  superiority.  The  Pacific  Fleet 
experienced  46  problems  with  the  NavyExtremely  High  Frequency  Satellite  Communications 
System  from  1995  to  2000.  The  problems  involved  hardware  and  software,  interoperability,  and 
training.  The  Pacific  Fleet  also  experienced  problems  with  the  High  Frequency  Radio  Group, 
mainly  due  to  system  performance  problems  and  training  shortfalls.  For  example,  on  a  ship  visit 
in  October  2000,  ship  communications  personnel  said  that  the  system  had  broken  down  several 
times  for  a  duration  of  1  week  to  1  month  at  a  time.  They  said  that,  when  it  breaks  down,  the 
operators  must  tune  in  the  radio  frequency  manually,  but  ship  operators  have  not  been  trained 
to  do  this  because  they  were  used  to  relying  on  the  system  to  tune  into  a  particular  frequency 
automatically. 


Entry  #528 

Presentation  To  The  V-22  Blue  Ribbon  Panel 
2001 
Navy 
V-22 

GAO-01-369R 

Quote  from  JAG  report:  “Aircraft  performance  envelopes  are  developed,  and  procedures  and 
guidance  published  (NATOPS)  to  prevent  pilots  from  putting  an  aircraft  in  a  situation  that  would 
exceed  safe  parameters.  The  MV-22  performance  envelope  may  be  one  that  fleet  pilots  can 
operate  within,  but  given  the  rigors  of  combat,  real  world  operations,  and  realistic  training  for 
both,  the  consequences  of  exceeding  this  particular  envelope  appears  to  be  excessively  grave 
(departure  from  controlled  flight  with  no  warning).’’ 
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Entry  #529 

TACTICAL  AIRCRAFT  Modernization  Rians  Wiii  Not  Reduce  Average  Age  of 

Aircraft 


2001 

Joint 

Aii  tacticai  aircraft 
GAO-01-163 

DOD  has  stated  that  as  aircraft  age,  they  tend  to  break  more  often,  take  ionger  to  inspect  and 
maintain,  and  are  iess  avaiiabie  for  operations  and  training. 


Entry  #530 

Assessments  of  Major  Weapon  Programs 

2003 

Joint 

JPATS  PAC-3 


JPATS:  They  aiso  reported  that  the  fuii  JPATS  had  not  yet  been  tested  due  to  uncorrected 
deficiencies  in  the  aircraft  and  the  immaturity  of  the  software-intensive  training  information 
management  system  (p.  48).  PAC-3:  However,  the  contractor  must  increase  production 
eariier  than  pianned  because  DOD  decided  to  acceierate  deiiveries.  This  decision  may  present 
new  production  chaiienges  because  the  contractor  must  find  and  train  additionai  personnei.  (p. 
55) 


Entry  #531 

DOD  Needs  to  Better  Manage  Automatic  Test  Equipment  Modernization 

2003 

Joint 

Consoiidated  Automated  Support  System 
GAO-03-451 

Counterexampie:  The  Navy  reports  that  the  repiacement  of  these  testers  with  CASS 
(Consoiidated  Automated  Support  System)  stations,  when  compiete,  wiii  reduce  the  number  of 
test-reiated  eniisted  occupationai  speciaities  from  32  to  4,  thus  reducing  training  requirements. 


Entry  #532 

TACTICAL  AIRCRAFT  Status  of  the  F/A-22  and  Joint  Strike  Fighter  Programs 

2004 

Joint 

F/A-22  and  Joint  Strike  Fighter 


If  the  Air  Force  faiis  to  improve  the  F/A-22’s  reiiabiiity  before  fieiding  the  aircraft,  the  high  faiiure 
rates  wiii  resuit  in  higher  operationai  and  support  costs  to  keep  the  aircraft  avaiiabie  for  training 
or  combat  use 

Entry  #533 
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MILITARY  AIRCRAFT  DOD  Needs  to  Determine  Its  Aerial  Refueling  Aircraft 
Requirements 
2004 

Air  Force 

KC-135 

GAO-04-349 

At  times,  Air  Force  refueling  aircraft  have  not  been  available  to  support  Navy  training  due  to  the 
lower  priority  placed  on  Navy  training  by  the  Air  Force,  and  sometimes  training  plans  changed 
on  relatively  short-notice 


Entry  #534 

DEFENSE  ACQUISITIONS  Assessments  of  Selected  Weapon  Programs 
2011 
Joint 


GAO-1 1-233SP 

The  Air  Force’s  C-130  AMP  will  standardize  the  cockpit  and  avionics  for  three  combat 
configurations  of  the  C-130  fleet.  In  February  2010  the  program  reported  a  Nunn-McCurdy 
unit  cost  breach  of  the  significant  cost  growth  threshold,  which  it  attributed  to  factors  such  as  the 
omission  of  training  devices  and  adequate  spares  from  initial  estimates,  and  delays  in  the 
production  decision.  The  program  has  been  restructured  and  planned  dates  for  key  events  have 
been  pushed  back  by  more  than  1  year.  (p.  51)  WIN-T  is  the  Army’s  high-speed  and  high- 
capacity  backbone  communications  network.  The  test  results  showed  that  WIN-T  Increment  2 
did  not  meet  its  operational  reliability  requirements.  DOD’s  Director,  Operational  Test  and 
Evaluation,  recommended  that  the  Army  improve  performance  and  training  to  address  these 
deficiencies  and  ensure  success  during  initial  operational  test,  which  is  scheduled  for  early  fiscal 
year  2012.  (p.  127) 


Entry  #535 

Realizing  Savings  under  Different  Littoral  Combat  Ship  Acquisition  Strategies  Depends  on 
Successful  Management  of  Risks 
2010 
Navy 
LCS 

GAO-1 1-277T 

Navy  officials  expressed  confidence  that  their  cost  estimate  supporting  the  dual  award  provides 
details  on  the  costs  to  operate  and  support  both  designs.  However,  since  little  actual  LCS 
operating  and  support  data  are  available  to  date,  the  Navy’s  estimates  for  these  costs  are 
currently  based  on  data  from  other  ships  and  could  change  as  actual  cost  data  become  more 
available.  These  estimates  are  also  based  on  new  operational  concepts  for  personnel,  training, 
and  maintenance  that  have  not  been  fully  developed,  tested,  and  implemented.  For  example, 
the  Navy  has  not  yet  implemented  a  comprehensive  training  plan,  and  it  is  possible  that  the  plan 
could  cost  more  or  less  than  the  training  costs  currently  accounted  for  by  the  Navy.  (p.  7) 
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